diffsync

DiffSync front-end classes and logic.

Copyright (c) 2020-2021 Network To Code, LLC info@networktocode.com

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

diffsync.Adapter

Class for storing a group of DiffSyncModel instances and diffing/synchronizing to another Adapter instance.

top_level = [] class-attribute

List of top-level modelnames to begin from when diffing or synchronizing.

type = None class-attribute instance-attribute

Type of the object, will default to the name of the class if not provided.

__init__(name=None, internal_storage_engine=LocalStore)

Generic initialization function.

Subclasses should be careful to call super().init() if they override this method.

__init_subclass__()

Validate that references to specific DiffSyncModels use the correct modelnames.

Called automatically on subclass declaration.

__len__()

Total number of elements stored.

__new__(**kwargs)

Document keyword arguments that were used to initialize Adapter.

__repr__()

Representation of an Adapter.

__str__()

String representation of an Adapter.

add(obj)

Add a DiffSyncModel object to the store.

Parameters:

Name	Type	Description	Default
obj	DiffSyncModel	Object to store	required

Raises:

Type	Description
ObjectAlreadyExists	if a different object with the same uid is already present.

count(model=None)

Count how many objects of one model type exist in the backend store.

Parameters:

Name	Type	Description	Default
model	Union[StrType, DiffSyncModel, Type[DiffSyncModel], None]	The DiffSyncModel to check the number of elements. If not provided, default to all.	None

Returns:

Type	Description
int	Number of elements of the model type

dict(exclude_defaults=True, **kwargs)

Represent the DiffSync contents as a dict, as if it were a Pydantic model.

diff_from(source, diff_class=Diff, flags=DiffSyncFlags.NONE, callback=None)

Generate a Diff describing the difference from the other DiffSync to this one.

Parameters:

Name	Type	Description	Default
source	Adapter	Object to diff against.	required
diff_class	Type[Diff]	Diff or subclass thereof to use for diff calculation and storage.	Diff
flags	DiffSyncFlags	Flags influencing the behavior of this diff operation.	NONE
callback	Optional[Callable[[StrType, int, int], None]]	Function with parameters (stage, current, total), to be called at intervals as the calculation of the diff proceeds.	None

diff_to(target, diff_class=Diff, flags=DiffSyncFlags.NONE, callback=None)

Generate a Diff describing the difference from this DiffSync to another one.

Parameters:

Name	Type	Description	Default
target	Adapter	Object to diff against.	required
diff_class	Type[Diff]	Diff or subclass thereof to use for diff calculation and storage.	Diff
flags	DiffSyncFlags	Flags influencing the behavior of this diff operation.	NONE
callback	Optional[Callable[[StrType, int, int], None]]	Function with parameters (stage, current, total), to be called at intervals as the calculation of the diff proceeds.	None

get(obj, identifier)

Get one object from the data store based on its unique id.

Parameters:

Name	Type	Description	Default
obj	Union[StrType, DiffSyncModel, Type[DiffSyncModel]]	DiffSyncModel class or instance, or modelname string, that defines the type of the object to retrieve	required
identifier	Union[StrType, Dict]	Unique ID of the object to retrieve, or dict of unique identifier keys/values	required

Raises:

Type	Description
ValueError	if obj is a str and identifier is a dict (can't convert dict into a uid str without a model class)
ObjectNotFound	if the requested object is not present

get_all(obj)

Get all objects of a given type.

Parameters:

Name	Type	Description	Default
obj	Union[StrType, DiffSyncModel, Type[DiffSyncModel]]	DiffSyncModel class or instance, or modelname string, that defines the type of the objects to retrieve	required

Returns:

Type	Description
List[DiffSyncModel]	List of Object

get_all_model_names()

Get all model names.

Returns:

Type	Description
Set[StrType]	List of model names

get_by_uids(uids, obj)

Get multiple objects from the store by their unique IDs/Keys and type.

Parameters:

Name	Type	Description	Default
uids	List[StrType]	List of unique id / key identifying object in the database.	required
obj	Union[StrType, DiffSyncModel, Type[DiffSyncModel]]	DiffSyncModel class or instance, or modelname string, that defines the type of the objects to retrieve	required

Raises:

Type	Description
ObjectNotFound	if any of the requested UIDs are not found in the store

get_or_add_model_instance(obj)

Attempt to get the object with provided obj identifiers or add obj.

Parameters:

Name	Type	Description	Default
obj	DiffSyncModel	An obj of the DiffSyncModel to get or add.	required

Returns:

Type	Description
Tuple[DiffSyncModel, bool]	Provides the existing or new object and whether it was created or not.

get_or_instantiate(model, ids, attrs=None)

Attempt to get the object with provided identifiers or instantiate and add it with provided identifiers and attrs.

Parameters:

Name	Type	Description	Default
model	Type[DiffSyncModel]	The DiffSyncModel to get or create.	required
ids	Dict	Identifiers for the DiffSyncModel to get or create with.	required
attrs	Optional[Dict]	Attributes when creating an object if it doesn't exist. Defaults to None.	None

Returns:

Type	Description
Tuple[DiffSyncModel, bool]	Provides the existing or new object and whether it was created or not.

get_or_none(obj, identifier)

Get one object from the data store based on its unique id or get a None.

Parameters:

Name	Type	Description	Default
obj	Union[StrType, DiffSyncModel, Type[DiffSyncModel]]	DiffSyncModel class or instance, or modelname string, that defines the type of the object to retrieve	required
identifier	Union[StrType, Dict]	Unique ID of the object to retrieve, or dict of unique identifier keys/values	required

Raises:

Type	Description
ValueError	if obj is a str and identifier is a dict (can't convert dict into a uid str without a model class)

Returns:

Type	Description
Optional[DiffSyncModel]	DiffSyncModel matching provided criteria

get_tree_traversal(as_dict=False) classmethod

Get a string describing the tree traversal for the diffsync object.

Parameters:

Name	Type	Description	Default
as_dict	bool	Whether to return as a dictionary	False

Returns:

Type	Description
Union[StrType, Dict]	A string or dictionary representation of tree

load()

Load all desired data from whatever backend data source into this instance.

load_from_dict(data)

The reverse of dict method, taking a dictionary and loading into the inventory.

Parameters:

Name	Type	Description	Default
data	Dict	Dictionary in the format that dict would export as	required

remove(obj, remove_children=False)

Remove a DiffSyncModel object from the store.

Parameters:

Name	Type	Description	Default
obj	DiffSyncModel	object to remove	required
remove_children	bool	If True, also recursively remove any children of this object	False

Raises:

Type	Description
ObjectNotFound	if the object is not present

str(indent=0)

Build a detailed string representation of this Adapter.

sync_complete(source, diff, flags=DiffSyncFlags.NONE, logger=None)

Callback triggered after a sync_from operation has completed and updated the model data of this instance.

Note that this callback is only triggered if the sync actually resulted in data changes. If there are no detected changes, this callback will not be called.

The default implementation does nothing, but a subclass could use this, for example, to perform bulk updates to a backend (such as a file) that doesn't readily support incremental updates to individual records.

Parameters:

Name	Type	Description	Default
source	Adapter	The DiffSync whose data was used to update this instance.	required
diff	Diff	The Diff calculated prior to the sync operation.	required
flags	DiffSyncFlags	Any flags that influenced the sync.	NONE
logger	Optional[BoundLogger]	Logging context for the sync.	None

sync_from(source, diff_class=Diff, flags=DiffSyncFlags.NONE, callback=None, diff=None)

Synchronize data from the given source DiffSync object into the current DiffSync object.

Parameters:

Name	Type	Description	Default
source	Adapter	object to sync data from into this one	required
diff_class	Type[Diff]	Diff or subclass thereof to use to calculate the diffs to use for synchronization	Diff
flags	DiffSyncFlags	Flags influencing the behavior of this sync.	NONE
callback	Optional[Callable[[StrType, int, int], None]]	Function with parameters (stage, current, total), to be called at intervals as the calculation of the diff and subsequent sync proceed.	None
diff	Optional[Diff]	An existing diff to be used rather than generating a completely new diff.	None

Returns:

Type	Description
Diff	Diff between origin object and source

Raises: DiffClassMismatch: The provided diff's class does not match the diff_class

sync_to(target, diff_class=Diff, flags=DiffSyncFlags.NONE, callback=None, diff=None)

Synchronize data from the current DiffSync object into the given target DiffSync object.

Parameters:

Name	Type	Description	Default
target	Adapter	object to sync data into from this one.	required
diff_class	Type[Diff]	Diff or subclass thereof to use to calculate the diffs to use for synchronization	Diff
flags	DiffSyncFlags	Flags influencing the behavior of this sync.	NONE
callback	Optional[Callable[[StrType, int, int], None]]	Function with parameters (stage, current, total), to be called at intervals as the calculation of the diff and subsequent sync proceed.	None
diff	Optional[Diff]	An existing diff that will be used when determining what needs to be synced.	None

Returns:

Type	Description
Diff	Diff between origin object and target

Raises: DiffClassMismatch: The provided diff's class does not match the diff_class

update(obj)

Update a DiffSyncModel object to the store.

Parameters:

Name	Type	Description	Default
obj	DiffSyncModel	Object to store	required

Raises:

Type	Description
ObjectAlreadyExists	if a different object with the same uid is already present.

update_or_add_model_instance(obj)

Attempt to update an existing object with provided obj ids/attrs or instantiate obj.

Parameters:

Name	Type	Description	Default
obj	DiffSyncModel	An instance of the DiffSyncModel to update or create.	required

Returns:

Type	Description
Tuple[DiffSyncModel, bool]	Provides the existing or new object and whether it was created or not.

update_or_instantiate(model, ids, attrs)

Attempt to update an existing object with provided ids/attrs or instantiate it with provided identifiers and attrs.

Parameters:

Name	Type	Description	Default
model	Type[DiffSyncModel]	The DiffSyncModel to update or create.	required
ids	Dict	Identifiers for the DiffSyncModel to update or create with.	required
attrs	Dict	Attributes when creating/updating an object if it doesn't exist. Pass in empty dict, if no specific attrs.	required

Returns:

Type	Description
Tuple[DiffSyncModel, bool]	Provides the existing or new object and whether it was created or not.

diffsync.DiffSync

Bases: Adapter

For backwards-compatibility, keep around the old name.

diffsync.DiffSyncModel

Bases: BaseModel

Base class for all DiffSync object models.

Note that read-only APIs of this class are implemented as get_*() methods rather than as properties; this is intentional as specific model classes may want to use these names (type, keys, attrs, etc.) as model attributes and we want to avoid any ambiguity or collisions.

This class has several underscore-prefixed class variables that subclasses should set as desired; see below.

NOTE: The groupings _identifiers, _attributes, and _children are mutually exclusive; any given field name can be included in at most one of these three tuples.

adapter = None class-attribute instance-attribute

Optional: the Adapter instance that owns this model instance.

model_config = ConfigDict(arbitrary_types_allowed=True) class-attribute instance-attribute

Pydantic-specific configuration to allow arbitrary types on this class.

model_flags = DiffSyncModelFlags.NONE class-attribute instance-attribute

Optional: any non-default behavioral flags for this DiffSyncModel.

Can be set as a class attribute or an instance attribute as needed.

__pydantic_init_subclass__(**kwargs) classmethod

Validate that the various class attribute declarations correspond to actual instance fields.

Called automatically on subclass declaration.

__repr__()

Return a string representation of this DiffSyncModel.

__str__()

Return a string representation of this DiffSyncModel.

add_child(child)

Add a child reference to an object.

The child object isn't stored, only its unique id. The name of the target attribute is defined in _children per object type

Raises:

Type	Description
ObjectStoreWrongType	if the type is not part of _children
ObjectAlreadyExists	if the unique id is already stored

create(adapter, ids, attrs) classmethod

Instantiate this class, along with any platform-specific data creation.

Subclasses must call super().create() or self.create_base(); they may wish to then override the default status information by calling set_status() to provide more context (such as details of any interactions with underlying systems).

Parameters:

Name	Type	Description	Default
adapter	Adapter	The master data store for other DiffSyncModel instances that we might need to reference	required
ids	Dict	Dictionary of unique-identifiers needed to create the new object	required
attrs	Dict	Dictionary of additional attributes to set on the new object	required

Returns:

Name	Type	Description
DiffSyncModel	Optional[Self]	instance of this class, if all data was successfully created.
None	Optional[Self]	if data creation failed in such a way that child objects of this model should not be created.

Raises:

Type	Description
ObjectNotCreated	if an error occurred.

create_base(adapter, ids, attrs) classmethod

Instantiate this class, along with any platform-specific data creation.

This method is not meant to be subclassed, users should redefine create() instead.

Parameters:

Name	Type	Description	Default
adapter	Adapter	The master data store for other DiffSyncModel instances that we might need to reference	required
ids	Dict	Dictionary of unique-identifiers needed to create the new object	required
attrs	Dict	Dictionary of additional attributes to set on the new object	required

Returns:

Name	Type	Description
DiffSyncModel	Optional[Self]	instance of this class.

create_unique_id(**identifiers) classmethod

Construct a unique identifier for this model class.

Parameters:

Name	Type	Description	Default
**identifiers	Dict[StrType, Any]	Dict of identifiers and their values, as in get_identifiers().	{}

delete()

Delete any platform-specific data corresponding to this instance.

Subclasses must call super().delete() or self.delete_base(); they may wish to then override the default status information by calling set_status() to provide more context (such as details of any interactions with underlying systems).

Returns:

Name	Type	Description
DiffSyncModel	Optional[Self]	this instance, if all data was successfully deleted.
None	Optional[Self]	if data deletion failed in such a way that child objects of this model should not be deleted.

Raises:

Type	Description
ObjectNotDeleted	if an error occurred.

delete_base()

Base delete method Delete any platform-specific data corresponding to this instance.

This method is not meant to be subclassed, users should redefine delete() instead.

Returns:

Name	Type	Description
DiffSyncModel	Optional[Self]	this instance.

dict(**kwargs)

Convert this DiffSyncModel to a dict, excluding the adapter field by default as it is not serializable.

get_attrs()

Get all the non-primary-key attributes or parameters for this object.

Similar to Pydantic's BaseModel.dict() method, with the following key differences: 1. Does not include the fields in _identifiers 2. Only includes fields explicitly listed in _attributes 3. Does not include any additional fields not listed in _attributes

Returns:

Name	Type	Description
dict	Dict	Dictionary of attributes for this object

get_children_mapping() classmethod

Get the mapping of types to fieldnames for child models of this model.

get_identifiers()

Get a dict of all identifiers (primary keys) and their values for this object.

Returns:

Name	Type	Description
dict	Dict	dictionary containing all primary keys for this device, as defined in _identifiers

get_shortname()

Get the (not guaranteed-unique) shortname of an object, if any.

By default the shortname is built based on all the keys defined in _shortname. If _shortname is not specified, then this function is equivalent to get_unique_id().

Returns:

Name	Type	Description
str	StrType	Shortname of this object

get_status()

Get the status of the last create/update/delete operation on this object, and any associated message.

get_type() classmethod

Return the type AKA modelname of the object or the class.

Returns:

Name	Type	Description
str	StrType	modelname of the class, used in to store all objects

get_unique_id()

Get the unique ID of an object.

By default the unique ID is built based on all the primary keys defined in _identifiers.

Returns:

Name	Type	Description
str	StrType	Unique ID for this object

json(**kwargs)

Convert this DiffSyncModel to a JSON string, excluding the adapter field by default as it is not serializable.

remove_child(child)

Remove a child reference from an object.

The name of the storage attribute is defined in _children per object type.

Raises:

Type	Description
ObjectStoreWrongType	if the child model type is not part of _children
ObjectNotFound	if the child wasn't previously present.

set_status(status, message='')

Update the status (and optionally status message) of this model in response to a create/update/delete call.

str(include_children=True, indent=0)

Build a detailed string representation of this DiffSyncModel and optionally its children.

update(attrs)

Update the attributes of this instance, along with any platform-specific data updates.

Subclasses must call super().update() or self.update_base(); they may wish to then override the default status information by calling set_status() to provide more context (such as details of any interactions with underlying systems).

Parameters:

Name	Type	Description	Default
attrs	Dict	Dictionary of attributes to update on the object	required

Returns:

Name	Type	Description
DiffSyncModel	Optional[Self]	this instance, if all data was successfully updated.
None	Optional[Self]	if data updates failed in such a way that child objects of this model should not be modified.

Raises:

Type	Description
ObjectNotUpdated	if an error occurred.

update_base(attrs)

Base Update method to update the attributes of this instance, along with any platform-specific data updates.

This method is not meant to be subclassed, users should redefine update() instead.

Parameters:

Name	Type	Description	Default
attrs	Dict	Dictionary of attributes to update on the object	required

Returns:

Name	Type	Description
DiffSyncModel	Optional[Self]	this instance.