How to write a new intel module#
If you want to add a new data type to Cartography, this is the guide for you. We look forward to receiving your PR!
Before getting started…#
Read through and follow the setup steps in the Cartography developer guide. Learn the basics of running, testing, and linting your code there.
The fast way#
To get started coding without reading this doc, use the AWS EMR implementation as an example. Its ingestion code contains the API and sync functions, while its declarative data model defines nodes and relationships. For a longer written explanation of the “how” and “why”, read on.
Configuration and credential management#
Supplying credentials and arguments to your module#
If you need to supply an API key or other credential to your Cartography module, we recommend adding a CLI argument. The CLI uses Typer with options organized into help panels per module. Add your options in cartography/cli.py following the existing patterns (e.g., Okta options). Credentials are typically read from environment variables and bound to Cartography’s Config object which is available in all modules.
An important note on validating your commandline args#
Note that it is your module’s responsibility to validate arguments that you introduce. For example with the Okta module, we validate that config.okta_api_key has been defined before attempting to continue.
Documenting module configuration#
Every intel module must provide docs/root/modules/<module>/config.md. Keep the
page focused on the steps required to run the module and use the following
canonical structure. Omit optional sections that do not apply instead of
leaving empty headings.
# <Module> Configuration
<!-- Briefly state what must be configured before this module can run. -->
## Prerequisites
<!-- Optional. List provider-side resources or tools that must already exist. -->
## Authentication
<!-- Required for API-backed modules. Explain how to create and supply credentials. -->
### <Authentication method>
<!-- Optional. Use subsections only when the module supports multiple methods. -->
## Required Permissions
<!-- Optional. Prefer a table for permissions. -->
## Optional Permissions
<!-- Optional. State which feature each permission enables. -->
## Configure Cartography
<!-- Required. Document environment variables, CLI options, and accepted values. -->
## Run Cartography
<!-- Required. Include at least one directly runnable command. -->
## Input Artifacts
<!-- Optional for report- or file-backed modules. -->
### Generate Input Artifacts
<!-- Optional. Explain how to create the artifacts Cartography consumes. -->
### Input Format
<!-- Optional. Document only setup-relevant format requirements. -->
## Advanced Configuration
<!-- Optional. Cover multi-account, multi-tenant, filtering, or alternate modes. -->
## Troubleshooting
<!-- Optional. Include only configuration-specific failures and remedies. -->
## References
<!-- Optional. Link to authoritative provider and Cartography documentation. -->
Use one H1 page title and start sections at H2. Order setup as prerequisites,
authentication, permissions, Cartography configuration, and a runnable command.
Store secrets in environment variables and clearly document *-env-var
indirection. Prefer tables for permissions or when documenting three or more
configuration options. Distinguish required permissions from optional
permissions and explain any graceful degradation.
Put module purpose, feature inventories, architecture, broad ingestion
behavior, and ontology integration in index.md. Put Cypher investigations in
queries.md or examples.md, and put post-ingestion analysis behavior in
analysis.md. Node, relationship, and property documentation belongs in model
docstrings and PropertyRef.description, which Sphinx uses to generate
schema.md.
Sync = Get, Transform, Load, Cleanup#
A cartography intel module consists of one sync function. sync should call get, then load, and finally cleanup.
Get#
The get function returns data as a list of dicts
from a resource provider API, which is GCP in this particular example.
get should be “dumb” in the sense that it should not handle retry logic or data
manipulation. It should also raise an exception if it’s not able to complete successfully.
Transform#
The transform function manipulates the list of dicts
to make it easier to ingest to the graph. transform functions are sometimes omitted when a module author decides that the output from the get is already in the shape that they need.
We have some best practices on handling transforms:
Handling required versus optional fields#
We should directly access dicts in cases where not having the data should cause a sync to fail.
For example, if we are transforming AWS data, we definitely need an AWS object’s ARN field because it uniquely
identifies the object. Therefore, we should access an object’s ARN using data['arn'] as opposed to
using data.get('arn') (the former will raise a KeyError if arn does not exist and the latter will just return
None without an exception).
We want the sync to fail if an important field is not present in our data. The idea here is that it is better to fail a sync than to add malformed data.
On the other hand, we should use data.get('SomeField') if SomeField is something optional that can afford to be
None.
For the sake of consistency, if a field does not exist, set it to None and not "".
Neo4j handles fields in datetime format, so when a date is returned as a string, it’s best to parse it to enable the use of operators during querying.
Load#
As seen in our AWS EMR ingestion code, the load function ingests a list of dicts to Neo4j by calling cartography.client.core.tx.load():
def load_emr_clusters(
neo4j_session: neo4j.Session,
cluster_data: List[Dict[str, Any]],
region: str,
current_aws_account_id: str,
aws_update_tag: int,
) -> None:
load(
neo4j_session,
EMRClusterSchema(),
cluster_data,
lastupdated=aws_update_tag,
Region=region,
AWS_ID=current_aws_account_id,
)
Tip
When defining nodes and properties, please follow the naming convention below:
Node classes should end with
SchemaRelationship classes should end with
RelNode property classes should end with
PropertiesRelationship property classes should end with
RelProperties
Defining a node#
As an example of a CartographyNodeSchema, you can view our EMRClusterSchema code:
@dataclass(frozen=True)
class EMRClusterSchema(CartographyNodeSchema):
label: str = 'AWSEMRCluster' # The label of the node
properties: EMRClusterNodeProperties = EMRClusterNodeProperties() # An object representing all properties on the EMR Cluster node
sub_resource_relationship: EMRClusterToAWSAccountRel = EMRClusterToAWSAccountRel()
An EMRClusterSchema object inherits from the CartographyNodeSchema class and contains a node label, properties, and a connection to its sub-resource: an AWSAccount.
Note that the typehints are necessary for Python dataclasses to work properly.
Defining node properties#
Here’s our EMRClusterNodeProperties code:
@dataclass(frozen=True)
class EMRClusterNodeProperties(CartographyNodeProperties):
arn: PropertyRef = PropertyRef('ClusterArn', extra_index=True)
firstseen: PropertyRef = PropertyRef('firstseen')
id: PropertyRef = PropertyRef('Id')
# ...
lastupdated: PropertyRef = PropertyRef('lastupdated', set_in_kwargs=True)
region: PropertyRef = PropertyRef('Region', set_in_kwargs=True)
security_configuration: PropertyRef = PropertyRef('SecurityConfiguration')
A CartographyNodeProperties object consists of PropertyRef objects. PropertyRefs tell querybuilder.build_ingestion_query() where to find appropriate values for each field from the list of dicts.
For example, id: PropertyRef = PropertyRef('Id') above tells the querybuilder to set a field called id on the AWSEMRCluster node using the value located at key 'Id' on each dict in the list. The PropertyRef argument is case-sensitive and names the source field, while the dataclass attribute names the Neo4j property.
As another example, region: PropertyRef = PropertyRef('Region', set_in_kwargs=True) tells the querybuilder to set a field called region on the AWSEMRCluster node using a keyword argument called Region supplied to cartography.client.core.tx.load(). set_in_kwargs=True is useful in cases where we want every object loaded by a single call to load() to have the same value for a given attribute.
Node property indexes#
Cartography uses its data model to automatically create indexes for
node properties that uniquely identify the node (e.g.
id)node properties are used to connect a node to other nodes (i.e. they are used as part of a
TargetNodeMatcheron aCartographyRelSchema.)a node’s
lastupdatedfield – this is used to enable faster cleanup jobs
As seen in the above definition for EMRClusterNodeProperties.arn, you can also explicitly specify additional indexes for fields that you expect to be queried on by providing extra_index=True to the PropertyRef constructor:
class EMRClusterNodeProperties(CartographyNodeProperties):
# ...
arn: PropertyRef = PropertyRef('ClusterArn', extra_index=True)
Index creation is idempotent (we only create them if they don’t exist).
See below for more information on indexes.
Extra node labels#
You can add additional Neo4j labels to a node with ExtraNodeLabels. This lets
different node types share a query target without changing their primary
labels. First define an uppercase ExtraNodeLabel constant in the appropriate
label module. Then import that constant into each node schema that needs it.
For example, AWS IAM users, groups, roles, and federated identities all use the
AWSPrincipal label so queries can address them as AWS principals:
from cartography.models.core.nodes import ExtraNodeLabel
# cartography/models/aws/extra_labels.py
AWS_PRINCIPAL = ExtraNodeLabel(
label="AWSPrincipal",
description="An AWS identity that can make authenticated requests.",
)
from cartography.models.aws.extra_labels import AWS_PRINCIPAL
from cartography.models.core.nodes import ExtraNodeLabels
from cartography.models.ontology.labels import PERMISSION_ROLE
@dataclass(frozen=True)
class AWSRoleSchema(CartographyNodeSchema):
label: str = "AWSRole"
properties: AWSRoleNodeProperties = AWSRoleNodeProperties()
extra_node_labels: ExtraNodeLabels = ExtraNodeLabels(
[AWS_PRINCIPAL, PERMISSION_ROLE]
)
This creates nodes with the labels
(:AWSRole:AWSPrincipal:PermissionRole). The role remains addressable by its
primary AWSRole label and can also participate in queries for AWS principals
or permission roles.
LabelKind.STANDARD is the default for provider-local and shared graph
interfaces. Use LabelKind.ONTOLOGY for cross-provider semantic labels. The
description field provides metadata for introspection and generated
documentation; it does not change ingestion behavior.
Define each label once and reuse its exported constant. Raw strings are not
accepted. Cross-provider ontology label constants live in
cartography.models.ontology.labels and use LabelKind.ONTOLOGY;
provider-local interface labels should remain near their provider models.
ExtraNodeLabels accepts an iterable and stores the labels as an immutable
tuple.
Compatibility labels#
Use a compatibility label only when renaming a node’s primary Neo4j label. It temporarily keeps the old label on newly ingested nodes so existing queries continue to work during the migration period. The node is still supported under its new primary label.
For example, CloudWatch log groups now use AWSCloudWatchLogGroup as their
primary label while retaining CloudWatchLogGroup until version 1.0.0:
from cartography.models.core.nodes import LabelKind
LEGACY_CLOUD_WATCH_LOG_GROUP = ExtraNodeLabel(
label="CloudWatchLogGroup",
description="Compatibility label for the former CloudWatchLogGroup node label.",
kind=LabelKind.COMPATIBILITY,
replacement_label="AWSCloudWatchLogGroup",
remove_in="1.0.0",
)
@dataclass(frozen=True)
class CloudWatchLogGroupSchema(CartographyNodeSchema):
label: str = "AWSCloudWatchLogGroup"
properties: CloudWatchLogGroupNodeProperties = CloudWatchLogGroupNodeProperties()
extra_node_labels: ExtraNodeLabels = ExtraNodeLabels(
[LEGACY_CLOUD_WATCH_LOG_GROUP]
)
replacement_label records the new primary label for introspection and
documentation. remove_in records the release in which the compatibility
label can be removed. Both fields are optional and are valid only when
kind=LabelKind.COMPATIBILITY.
Conditional node labels#
Warning
Conditional labels are a specialized feature primarily used for ontology mapping scenarios where a single data source produces records that map to different semantic types. Most intel modules do not need this feature.
Sometimes the label depends on values in each data dictionary processed by the
loader. Compose a conditional value from an exported constant with
CONSTANT.when(property_name="value"). The keyword must name a property in
the node schema. The loader sets that node property from its PropertyRef in
each input data dictionary before the condition is evaluated.
For example, the ECR transform
sets the type key on each image.
The
ECR image schema uses that key
to select the matching ontology label:
from cartography.models.core.nodes import ExtraNodeLabels
from cartography.models.ontology.labels import IMAGE
from cartography.models.ontology.labels import IMAGE_ATTESTATION
from cartography.models.ontology.labels import IMAGE_MANIFEST_LIST
@dataclass(frozen=True)
class ECRImageSchema(CartographyNodeSchema):
label: str = "AWSECRImage"
properties: ECRImageNodeProperties = ECRImageNodeProperties()
sub_resource_relationship: ECRImageToAccountRel = ECRImageToAccountRel()
# Apply different ontology labels based on the image type
extra_node_labels: ExtraNodeLabels = ExtraNodeLabels([
IMAGE.when(type="image"),
IMAGE_ATTESTATION.when(type="attestation"),
IMAGE_MANIFEST_LIST.when(type="manifest_list"),
])
Primary use case: Container Registry Images
ECR (and other container registries) store different types of artifacts that share the same base schema but have fundamentally different semantic meanings:
|
Ontology Label |
Description |
|---|---|---|
|
|
Standard container image |
|
|
SLSA/Sigstore attestation |
|
|
Multi-arch manifest list |
Without conditional labels, we cannot accurately map these to distinct ontology
types. An AWSECRImage node with type: "attestation" should be labeled as
ImageAttestation in the ontology, not just generic Image. Production ECR
values are image, attestation, and manifest_list. Exact matching is
case-sensitive, so uppercase variants do not match.
How it works:
Labels with empty
conditionsare applied unconditionally in the ingestion query’sSETclauseLabels with nonempty
conditionsare applied row by row in the same ingestion query, with a pair ofFOREACHclauses that add the label when the conditions hold and remove it when they do notwhen()returns a new immutable label value and leaves the exported constant unchangedConditions are stored as immutable, sorted
(field, value)tuplesConditions use case-sensitive exact string equality and are combined with AND logic
When conditions change, labels are automatically added or removed on subsequent syncs
Important notes:
Condition values must be strings (e.g.,
"true"notTrue)Condition field names must exist on the concrete node’s properties schema
All conditions in one
when()call must match for the label to be applied (AND logic). Declaring the same label more than once with different conditions is allowed: the label applies if any of the declarations matches (OR logic)Because labels are applied per row, only the nodes in the current batch are relabeled. A node whose conditions no longer hold is corrected the next time it is loaded, or deleted by cleanup if it is no longer reported
Indexes are automatically created for conditional labels themselves. Condition fields are not indexed: they are evaluated against the already-bound node of the current row, so there is no lookup for an index to serve
Defining relationships#
Relationships can be defined on CartographyNodeSchema through its sub_resource_relationship field or its other_relationships field (you can find an example of other_relationships here in our test data).
As seen above, an EMRClusterSchema only has a single relationship defined: an EMRClusterToAWSAccountRel:
@dataclass(frozen=True)
# (:AWSEMRCluster)<-[:RESOURCE]-(:AWSAccount)
class EMRClusterToAWSAccountRel(CartographyRelSchema):
target_node_label: str = 'AWSAccount' # (1)
target_node_matcher: TargetNodeMatcher = make_target_node_matcher( # (2)
{'id': PropertyRef('AccountId', set_in_kwargs=True)},
)
direction: LinkDirection = LinkDirection.INWARD # (3)
rel_label: str = "RESOURCE" # (4)
properties: EMRClusterToAWSAccountRelRelProperties = EMRClusterToAWSAccountRelRelProperties() # (5)
This class is best described by explaining how it is processed: build_ingestion_query() will traverse the EMRClusterSchema to its sub_resource_relationship field and find the above EMRClusterToAWSAccountRel object. With this information, we know to
draw a relationship to an
AWSAccountnode (1) using the label “RESOURCE” (4)by matching on the AWSAccount’s “
id” field” (2)where the relationship directionality is pointed inward toward the AWSEMRCluster (3)
making sure to define a set of properties for the relationship (5). The full example RelProperties is very short:
@dataclass(frozen=True)
class EMRClusterToAWSAccountRelRelProperties(CartographyRelProperties):
lastupdated: PropertyRef = PropertyRef('lastupdated', set_in_kwargs=True)
Important
Relationship Naming Guidelines
When naming relationships in Cartography:
Prefer clear verbs (e.g., OWNS, CONTAINS)
Avoid ambiguous or passive phrasing (e.g., IS, CAN)
Use direct and active forms
Prefer OWNS over OWNED_BY
Prefer CONTAINS over BELONGS_TO
Consistent, action-oriented naming improves graph readability and makes Cypher queries more intuitive.
Sub-Resources relationship#
A sub-resource is a specific type of composition relationship in which a node “belongs to” a higher-level entity such as an Account, Subscription, etc.
Examples:
In AWS, the parent is typically an
AWSAccount.In Azure, it’s a
TenantorSubscription.In GCP, it’s a
GCPProject.
To define a sub-resource relationship, use the sub_resource_relationship property on the node class. It must follow these constraints:
The target node matcher must have
set_in_kwargs=True(required for auto-cleanup functionality).All
sub_resource_relationships must:Use the label
RESOURCEHave the direction set to
INWARD
Each module:
Must have at least one root node (a node without a
sub_resource_relationship)Must have at most one root node
Common Relationship Types#
While you’re free to define custom relationships, using standardized types improves maintainability and facilitates querying and analysis.
Composition
(:Parent)-[:CONTAINS]->(:Child)(:Parent)-[:HAS]->(:Child)
Tagging
(:Entity)-[:TAGGED]->(:Tag)
Group Membership
(:Element)-[:MEMBER_OF]->(:Group)(:Element)-[:ADMIN_OF]->(:Group)Note
If an element is an admin, both relationships (
MEMBER_OFandADMIN_OF) should be present for consistency.
Ownership
(:Entity)-[:OWNS]->(:OtherEntity)
Permissions (ACL)
(:Actor)-[:CAN_ACCESS]->(:Entity)(:Actor)-[:CAN_READ]->(:Entity)(:Actor)-[:CAN_WRITE]->(:Entity)(:Actor)-[:CAN_ADD]->(:Entity)(:Actor)-[:CAN_DELETE]->(:Entity)
The result#
And those are all the objects necessary for this example! The resulting query will look something like this:
UNWIND $DictList AS item
MERGE (i:AWSEMRCluster{id: item.Id})
ON CREATE SET i.firstseen = timestamp()
SET
i.lastupdated = $lastupdated,
i.arn = item.ClusterArn
// ...
WITH i, item
CALL {
WITH i, item
OPTIONAL MATCH (j:AWSAccount{id: $AccountId})
WITH i, item, j WHERE j IS NOT NULL
MERGE (i)<-[r:RESOURCE]-(j)
ON CREATE SET r.firstseen = timestamp()
SET
r.lastupdated = $lastupdated
}
And that’s basically all you need to know to understand how to define your own nodes and relationships using cartography’s data objects. For more information, view the model API documentation as a reference.
Additional concepts#
This section explains cartography general patterns, conventions, and design decisions.
cartography’s update_tag:#
cartography’s global config object carries around an update_tag property which is a tag/label associated with the current sync. Cartography’s sync code sets this to a Unix timestamp of when the sync was started.
All cartography intel modules set the lastupdated property on all nodes and all relationships to this update_tag.
All nodes need these fields#
id - an ID should be a string that uniquely identifies the node. In AWS, this is usually an ARN. In GCP, this is usually a partial URI.
If possible, we should use API-provided fields for IDs and not create our own. In some cases though this is unavoidable - see GCPNetworkTag.
When setting an
id, ensure that you also include the field name that it came from. For example, since we’ve decided to usepartial_uris as an id for a GCPVpc, we should include bothpartial_uriandidon the node. This way, a user can tell what fields were used to derive theid. This is accomplished herelastupdated- See below on how this gets set automatically.firstseen- See below on how this gets set automatically.
All relationships need these fields#
Keep relationships lightweight unless an ingestion or query path requires additional properties. Standard relationships typically contain these two fields:
lastupdated- See below on how this gets set automatically.firstseen- See below on how this gets set automatically.
For MatchLinks loaded with load_matchlinks(), Cartography also creates a composite relationship index on _sub_resource_label and _sub_resource_id. Those stable keys support scoped cleanup; lastupdated is deliberately excluded because it changes on every sync.
Run queries only on indexed fields for best performance#
In this older example of ingesting GCP VPCs, we connect VPCs with GCPProjects
based on their id fields.
ids are indexed, as seen here
and here.
All of these queries use indexes for faster lookup.
indexes.cypher#
Older intel modules define indexes in indexes.cypher. By using CartographyNodeSchema and CartographyRelSchema objects, indexes are automatically created so you don’t need to update this file!
lastupdated and firstseen#
On every cartography node and relationship, we set the lastupdated field to the UPDATE_TAG and firstseen field to timestamp() (a built-in Neo4j function equivalent to epoch time in milliseconds). This is automatically handled by the cartography object model.
One-to-many relationships#
We can use the Cartography data model to represent one-to-many relationships. For example, an AWS IAM instance profile (API docs) maps to one or more roles.
An example instance profile object looks like this:
INSTANCE_PROFILES = [
{
"Path": "/",
"InstanceProfileName": "my-instance-profile",
"InstanceProfileId": "AIPA4SD",
"Arn": "arn:aws:iam::1234:instance-profile/my-instance-profile",
"CreateDate": datetime.datetime(2024, 12, 21, 23, 54, 16),
"Roles": [
{
"Path": "/",
"RoleName": "role1",
"RoleId": "AROA4",
"Arn": "arn:aws:iam::1234:role/role1",
"CreateDate": datetime.datetime(2024, 12, 21, 6, 53, 29),
},
{
"Path": "/",
"RoleName": "role2",
"RoleId": "AROA5",
"Arn": "arn:aws:iam::1234:role/role2",
"CreateDate": datetime.datetime(2024, 12, 21, 6, 53, 29),
},
],
},
]
Note that the Roles field in this data object is a list of objects (and that this is a one-to-many setup).
Here’s how to represent this in the Cartography data model:
Transform the data so that
Rolesbecomes a list of IDs and not dicts. Here we will use ARNs. The result should be:TRANSFORMED_INSTANCE_PROFILES = [ { "Path": "/", "InstanceProfileName": "my-instance-profile", "InstanceProfileId": "AIPA4SD", "Arn": "arn:aws:iam::1234:instance-profile/my-instance-profile", "CreateDate": datetime.datetime(2024, 12, 21, 23, 54, 16), "Roles": [ "arn:aws:iam::1234:role/role1", "arn:aws:iam::1234:role/role2", ] }, ]
Define the InstanceProfile node (irrelevant fields omitted for brevity):
@dataclass(frozen=True) class InstanceProfileSchema(CartographyNodeSchema): label: str = 'AWSInstanceProfile' properties: ... sub_resource_relationship: ... other_relationships: OtherRelationships = OtherRelationships([ InstanceProfileToAWSRoleRel(), ])
Define its association with AWS roles
@dataclass(frozen=True) class InstanceProfileToAWSRoleRel(CartographyRelSchema): target_node_label: str = 'AWSRole' target_node_matcher: TargetNodeMatcher = make_target_node_matcher( {'arn': PropertyRef('Roles', one_to_many=True)}, ) direction: LinkDirection = LinkDirection.OUTWARD rel_label: str = "ASSOCIATED_WITH" properties: ...
The key part is setting
one_to_many=Truein the PropertyRef for the TargetNodeMatcher. This instructs the data model to look for AWSRoles in the graph where theirarnfield is in the list pointed to by theRoleskey on the data dict.
Now we can use the same steps described above in this doc to finish data ingestion.
MatchLinks#
See the MatchLinks documentation on how to connect existing nodes in the graph together using Cartography’s data model.
Cleanup#
We have just added new nodes and relationships to the graph, and we have also updated previously-added ones
by using MERGE. We now need to delete nodes and relationships that no longer exist, and we do this by removing
all nodes and relationships that have lastupdated NOT set to the update_tag of this current run.
By using Cartography schema objects, a cleanup function is trivial to write:
def cleanup(neo4j_session: neo4j.Session, common_job_parameters: Dict) -> None:
logger.debug("Running EMR cleanup job.")
cleanup_job = GraphJob.from_node_schema(EMRClusterSchema(), common_job_parameters)
cleanup_job.run(neo4j_session)
Scoped cleanups#
By default, a node_schema has scoped_cleanup flag set to True. This means that when we run a clean up job on that
node type, then we will only delete stale nodes that are connected to the current sub-resource being synced. This is
designed for modules like AWS or GCP where there a clear definition of a “tenant”-like object because each account or
project gets synced in one at a time and it doesn’t make sense to delete objects outside of the current tenant being
synced.
For some other modules that don’t have a clear tenant-like relationship, you can set scoped_cleanup to False on the
node_schema. This might make sense for a vuln scanner module where there is no logical tenant object.
Hierarchical data and cascade_delete#
Some data sources have multi-tier hierarchical structures where nodes own other nodes via RESOURCE relationships. Examples include:
GCP: Organization → Folders → Projects → Compute instances, Storage buckets, etc.
GitLab: Organization → Groups → Projects → Branches, Dependencies, etc.
In Cartography, RESOURCE relationships point from parent to child:
(Parent)-[:RESOURCE]->(Child)
When a parent node becomes stale and is deleted, you may want its children to be deleted as well. The cascade_delete parameter enables this behavior:
def cleanup(neo4j_session: neo4j.Session, common_job_parameters: Dict) -> None:
cleanup_job = GraphJob.from_node_schema(
MyParentSchema(),
common_job_parameters,
cascade_delete=True, # Also delete children when parent is stale
)
cleanup_job.run(neo4j_session)
When cascade_delete=True, the cleanup query becomes:
WHERE n.lastupdated <> $UPDATE_TAG
WITH n LIMIT $LIMIT_SIZE
OPTIONAL MATCH (n)-[:RESOURCE]->(child)
WHERE child IS NULL OR child.lastupdated <> $UPDATE_TAG
DETACH DELETE child, n;
When to use cascade_delete:
Use
cascade_delete=Truewhen child nodes are meaningless without their parent (e.g., GitLab branches without their project)Use
cascade_delete=False(default) when children should persist independently or when another module manages their lifecycle
Important notes:
Only affects direct children (one level deep via
RESOURCErelationships). Grandchildren require cleaning up intermediate levels first.Children that were re-parented in the current sync (matching
UPDATE_TAG) are protected from deletion.Only valid with scoped cleanup (
scoped_cleanup=True). Unscoped cleanups will raise an error ifcascade_delete=True.Default is
Falsefor backward compatibility.
Legacy notes#
Older intel modules still do this process with hand-written cleanup jobs that work like this:
Delete all old nodes
You can see this in our GCP VPCs example. We run
DETACH DELETEto delete an old node and disconnect it from all other nodes.Delete all old relationships
You can see this in the GCP VPC example here and here.
Q: We just
DETACH DELETE’d the node. Why do we need to delete the relationships too?A: There are cases where the node may continue to exist but the relationships between it and other nodes have changed. Explicitly deleting stale relationships accounts for this case. See this short discussion.
Error handling principles#
Don’t catch the base Exception class when error handling because it makes problems difficult to trace.
Do catch the narrowest possible class of exception.
Only catch exceptions when your code can resolve the issue. Otherwise, allow exceptions to bubble up.
Schema documentation#
Do not create or edit a module’s schema.md manually. Sphinx generates schema
pages from the declarative data model. Add a docstring to every node and
relationship schema, and add a human-readable description= to every displayed
PropertyRef. Update these model definitions whenever the graph schema
changes.
Making tests#
Before making tests, read through and follow the setup steps in the Cartography developer guide.
Add fake data for testing at
tests/data. We can see the AWS EC2 instance example here.If needed, add unit tests to
tests/unit/cartography/intel. As seen in this GCP example, these tests ensure thattransform*manipulates the data in expected ways.Add integration tests to
tests/integration/cartography/intel. See this AWS EC2 instance example. By default, integration tests start a disposable Neo4j test container and verify that the loaded nodes and relationships match the mock data. Docker must be running. To use an existing Neo4j instance instead, setNEO4J_URL; integration tests delete all nodes from that database.
Other#
We prefer and will accept PRs which incrementally add information from a particular data source. Incomplete representations are OK provided they are consistent over time. For example, we don’t sync 100% of AWS resources but the resources that exist in the graph don’t change across syncs.
Each intel module offers its own view of the graph
Note
This best practice is a little less precise, so if you’ve gotten to this point and you need clarification, just submit your PR and ask us.
As much as possible, each intel module should ingest data without assuming that a different module will ingest the same data. Explained another way, each module should “offer its own perspective” on the data. We believe doing this gives us a more complete graph. Below are some key guidelines clarifying and justifying this design choice.
It is possible (and encouraged) for more than one intel module to modify the same node type. However, there are two distinct patterns for this:
Simple Relationship Pattern: When data type A only refers to data type B by an ID without providing additional properties about B, we can just define a relationship schema. This way when A is loaded, the relationship schema performs a
MATCHto find and connect to existing nodes of type B.For example, when an RDS instance refers to EC2 security groups by ID, we create a relationship from the RDS instance to the security group nodes, since the RDS API doesn’t provide additional properties about the security groups beyond their IDs.
# RDS Instance refers to Security Groups by ID only @dataclass(frozen=True) class RDSInstanceToSecurityGroupRel(CartographyRelSchema): target_node_label: str = "AWSEC2SecurityGroup" target_node_matcher: TargetNodeMatcher = make_target_node_matcher({ "id": PropertyRef("SecurityGroupId"), # Just the ID, no additional properties }) direction: LinkDirection = LinkDirection.OUTWARD rel_label: str = "MEMBER_OF_EC2_SECURITY_GROUP" properties: RDSInstanceToSecurityGroupRelProperties = RDSInstanceToSecurityGroupRelProperties()
Composite Node Pattern: When a data type
Arefers to another data typeBand offers additional fields aboutBthatBdoesn’t have itself, we should define a composite node schema. This composite node would be named “BASchema” to denote that it’s a “B” object as known by an “A” object. When loaded, the composite node schema targets the same node label as the primaryBschema, allowing the loading system to perform aMERGEoperation that combines properties from both sources.For example, in the AWS EC2 module, we have both
EBSVolumeSchema(from the EBS API) andEBSVolumeInstanceSchema(from the EC2 Instance API). The EC2 Instance API provides additional properties about EBS volumes that the EBS API doesn’t have, such asdeleteontermination. Both schemas target the sameAWSEBSVolumenode label, allowing the node to accumulate properties from both sources.# EC2 Instance provides additional properties about EBS Volumes @dataclass(frozen=True) class EBSVolumeInstanceProperties(CartographyNodeProperties): id: PropertyRef = PropertyRef("VolumeId") arn: PropertyRef = PropertyRef("Arn", extra_index=True) lastupdated: PropertyRef = PropertyRef("lastupdated", set_in_kwargs=True) # Additional property that EBS API doesn't have deleteontermination: PropertyRef = PropertyRef("DeleteOnTermination") @dataclass(frozen=True) class EBSVolumeInstanceSchema(CartographyNodeSchema): label: str = "AWSEBSVolume" # Same label as EBSVolumeSchema properties: EBSVolumeInstanceProperties = EBSVolumeInstanceProperties() sub_resource_relationship: EBSVolumeToAWSAccountRel = EBSVolumeToAWSAccountRel() # ... other relationships
The key distinction is whether the referring module provides additional properties about the target entity. If it does, use a composite node schema. If it only provides IDs, use a simple relationship schema.
In case you’re curious, here’s some historical context on how we got here.