Superdesk CLI

In superdesk/server folder you can find manage.py file which you can use to run commands from CLI:

$ python manage.py

With no other argument it will output list of all available commands. To run specific command you use its name and any params needed:

$ python manage.py users:create -u admin -p admin

Creating new command

You can create new commands using the superdesk.commands.cli attribute (which is an instance of superdesk.core.cli.AsyncAppGroup).:

import click
from superdesk.commands import cli

@cli.command("hello:world")
@click.option("-n", "--name", default="world", help="Name of the person to say hello to")
def cli_hello_world(name: str) -> None:
    print(f"Hello, {name}!")

To have the new command registered, make sure the command function is imported from one of the following settings:

• CORE_APPS

• INSTALLED_APPS

• MODULES

Simply importing of the file on app startup will include the command with the application.

These commands support both sync and async functions, no special changes are required.

We use Click under the hood so you can get more info there.

Superdesk commands

app:clean_images

cli_clean_images()

This command will remove all the images from the system which are not referenced by content.

It checks the media type and calls the correspoinding function as s3 and mongo requires different approaches for handling multiple files. Probably running db.repairDatabase() is needed in Mongo to shring the DB size.

Example:

$ python manage.py app:clean_images

app:deleteArchivedDocument

cli_delete_archived_document()

Deletes a Text Archive document(s) from both Mongodb and ElasticSearch.

It deletes digital package and the story given by id. It accepts one or more ids separated by space.

Example:

$ manage.py app:deleteArchivedDocument 588c1df11d41c80928015601 588c1b901d41c805dce70df0

app:index_from_mongo

cli_index_from_mongo()

Index the specified mongo collection in the specified elastic collection/type.

This will use the default APP mongo DB to read the data and the default Elastic APP index.

Use -f all to index all collections.

Example:

$ python manage.py app:index_from_mongo --from=archive
$ python manage.py app:index_from_mongo --all
$ python manage.py app:index_from_mongo --from=archive --date=2024-01-01
$ python manage.py app:index_from_mongo --all --date="2024-01-01T12:00:00"

app:initialize_data

app_initialize_data_command()

Initialize application with predefined data for various entities.

Loads predefined data (vocabularies, desks, etc..) for instance. Mostly used for to load initial data for production instances, using app:prepopulate command in this case is bad practice, because it will load a lot of redundant data which is difficult to get rid of.

Supported entities:

roles, users, desks, stages, vocabularies, validators, content_templates,
content_types, published, activity, archive, archive_versions, ingest,
publish_queue, archived, legal_archive, legal_archive_versions, legal_publish_queue,
dictionaries, ingest_providers, search_providers, products, subscribers,
workspaces, item_comments, audit, contacts, planning_types

If no –entity-name parameter is supplied, all the entities are inserted.

The entities:

vocabularies, validators, content_types, dictionaries, ingest_providers,
search_providers, products, subscribers, workspaces, item_comments,
planning_types

will be updated with the predefined data if it already exists, no action will be taken for the other entities.

Args:

entity_name (str | list | None, optional): Entity(ies) to initialize. path (str | None, optional): Path of the file to import. sample_data (bool, optional): If True, sample data will be used. force (bool, optional): If True, update items even if they have been modified by the user. init_index_only (bool, optional): If True, only the indexes are initialized.

Example:

$ python manage.py app:initialize_data
$ python manage.py app:initialize_data --entity-name=vocabularies
$ python manage.py app:initialize_data --entity-name=content_types

app:flush_elastic_index

flush_elastic_index_command()

Flush elastic index.

It removes elastic index, creates a new one and index it from mongo. You must specify at least one elastic index to flush: --sd (superdesk) or --capi (content api)

app:prepopulate

cli_app_prepopulate()

Prepopulate Superdesk using sample data.

Useful for demo/development environment, but don’t run in production, it’s hard to get rid of such data later.

Example:

$ python manage.py app:prepopulate

app:populate

cli_app_populate()

Insert or update data in collection using sample file.

Specified file must be a valid json file. Filename will be used as a collection name where data will be inserted or updated. If document already exists in the collection, it will be updated.

Example:

$ python manage.py app:populate --filepath=data/content_types.json
$ python manage.py app:populate --filepath=data/planning_types.json
$ python manage.py app:populate --filepath=data/vocabularies.json

app:rebuild_elastic_index

cli_rebuild_elastic_index()

Rebuild the elastic indexes from existing data.

It creates new index with same alias as the configured index, puts the new mapping and removes the old index.

Example:

$ python manage.py app:rebuild_elastic_index
$ python manage.py app:rebuild_elastic_index --resource=items
$ python manage.py app:rebuild_elastic_index --resource=archive

app:run_macro

run_macro()

Executes a macro by given name and optional keyword arguments.

Example:

$ app:run_macro --name clean_keywords --kwargs {"repo":"archived"}

app:scaffold_data

scaffold_data_command()

app:updateArchivedDocument

cli_update_archived_document()

Update metadata for a document in both Mongodb and ElasticSearch by supplying ids field to update and the value for the update

--field

Field name

--value

Value to be set in the field

--parseNeeded

Optional. True if value is a complex type, not an int or string

Example:

$ manage.py app:updateArchivedDocument --ids='["588c1df11d41c80928015601","588c1b901d41c805dce70df0"]'
    --field=anpa_category
    --value=[{"scheme" : null,"qcode" : "f","subject" : "04000000","name" : "Finance"}]
    --parseNeeded=True

archive:remove_expired

cli_archive_remove_expired()

Remove expired content from Superdesk.

It removes expired items from production, published and archived colections.

Example:

$ python manage.py archive:remove_expired

audit:purge

cli_audit_purge()

Purge audit collection.

Entries are purged if the related item is no longer in ‘Production’. Other entries are deleted after the configured time period.

Example:

$ python manage.py audit:purge
$ python manage.py audit:purge --expiry_minutes=20

content_api:remove_expired

cli_content_api_remove_expired()

Remove expired items from the content_api items collection.

By default no items expire there, you can change it using CONTENT_API_EXPIRY_DAYS config.

Example:

$ python manage.py content_api:remove_expired

data:generate_update

cli_data_generate_update()

Generate a file where to define a new data update.

Example:

$ python manage.py data:generate_update --resource=archive

data:upgrade

upgrade_command()

Runs all the new data updates available.

If data_update_id is given, runs new data updates until the given one.

Example:

$ python manage.py data:upgrade

data:downgrade

downgrade_command()

Runs the latest data update backward.

If data_update_id is given, runs all the data updates backward until the given one.

Example:

$ python manage.py data:downgrade

ingest:clean_expired

cli_ingest_clean_expired()

Remove stale data from ingest based on the provider settings.

Example:

$ python manage.py ingest:clean_expired
$ python manage.py ingest:clean_expired --provider=aap

ingest:provider

cli_add_provider()

Add ingest provider.

Example:

$ python manage.py ingest:provider --provider='{"update_schedule" : { "minutes" : 5, "seconds" : 0 },
    "idle_time" : { "hours" : 0, "minutes" : 0 }, "content_expiry" : 2880, "name" : "aap-demo",
     "source" : "aap-demo", "feeding_service" : "rss",
     "config" : { "url" : "https://abcnews.go.com/abcnews/primetimeheadlines", "field_aliases" : [ ] },
     "feed_parser" : null, "content_types" : [ "text" ]}'

ingest:update

cli_update_ingest()

Runs update for ingest providers.

Example:

$ python manage.py ingest:update
$ python manage.py ingest:update --provider=aap-demo

legal_archive:import

cli_legal_archive_import()

This command import archive into legal archive.

As per the publishing logic the import to legal archive is done asynchronously. If this fails then you are missing records in legal archive. Use this command to manually import archive items into legal archive.

Example:

$ python manage.py legal_archive:import
$ python manage.py legal_archive:import --page-size=100

legal_publish_queue:import

cli_legal_publish_queue_import()

This command import publish queue records into legal publish queue.

Example:

$ python manage.py legal_publish_queue:import
$ python manage.py legal_publish_queue:import --page-size=100

publish:enqueue

publish_scheduled_items()

Command to transmit and publish all scheduled items (if the schedule has elapsed).

This asynchronous function fetches the exchange factory and initiates the processing of pending tasks. Useful for ensuring any delayed or queued items are published as required.

publish:transmit

publish_pending_items()

Command to transmit and publish all pending items.

This asynchronous function fetches the exchange factory and initiates the processing of pending tasks. Useful for ensuring any delayed or queued items are published as required.

session:gc

cli_session_gc()

Remove expired sessions from db.

Using SESSION_EXPIRY_MINUTES config.

Example:

$ python manage.py session:gc

schema:migrate

schema_migrate_command()

Migrate elastic schema if needed, should be triggered on every deploy.

It compares version set in code (latest) to one stored in db and only updates schema if those are different.

Current version is read from settings and fallbacks to superdesk.SCHEMA_VERSION, so that you can avoid migration via settings file if needed.

Example:

$ python manage.py schema:migrate

Storage Dumps and Records

Note

You can check Database Recording for details on how to use following storage:… commands.

cli_data_storage_dump(*args: t.Any, **kwargs: t.Any) → t.Any

Dump collections from MongoDB

Note: this command should only be used for development purpose. Use MongoDB official commands to dump or restore databases in production.

Dump whole collection into either separate JSON files (one per collection) or a single one (with a single root object).

Single file are easier to copy, while separate JSON files are easier to check.

A dump is full Superdesk save, as opposed to a record which only store change made in a database (more like a diff).

Example: Do a full database dump with a name and description to a single file:

$ python manage.py storage:dump -n "demo-instance" -D "this dump includes some test data (desks, users) to run a
basic demo of Superdesk" --single

cli_data_storage_restore(*args: t.Any, **kwargs: t.Any) → t.Any

Restore MongoDB collections dumped with storage:dump

Example:

$ storage:restore foobar_superdesk_dump

cli_data_storage_record(*args: t.Any, **kwargs: t.Any) → t.Any

Record changes made in database until the command is stopped

This command is intended for developers to help producing specific state (e.g. for tests), or to create a specific Superdesk instance e.g. for a demo.

If you specify a base dump (with --base-dump), the database will be restored to this dump before starting the record, and it will be associated with the record.

If no base dump is specified, the database should be in the same state as when the record has been done when you restore it.

If you use the --full-document option, the whole document will be stored in the record in case of update (instead of just diff), this will result in a bigger dump file, but may be applied to a database even if it was not exactly in the same state as when the record has been done.

You may want to use --collection to record change only on the collections you’re interested in, and avoir side effects.

Example: Record change in vocabularies only, with a name and description, and base on “base_test_e2e_dump” dump:

$ python manage.py storage:record -b "base_test_e2e_dump" -c vocabularies -n "test_categories"-D "prepare
  instance for categories end-to-end tests"

cli_data_storage_restore_record(*args: t.Any, **kwargs: t.Any) → t.Any

Restore Superdesk record

This command is to be used with a record dump, not a full database archive.

Example:

$ storage:restore-record record-for-some-e2e-test

cli_data_storage_list(*args: t.Any, **kwargs: t.Any) → t.Any

List Superdesk Dumps and Records

cli_data_storage_upgrade_dumps(*args: t.Any, **kwargs: t.Any) → t.Any

Apply migration scripts on all dumps and records

Note: the backend MUST NOT be running while this command is used.

Migration scripts are first applied on each records: changes on collections used in the record are merged to the record.

Then for full dump, migration are normally applied and dumps are updated.

The whole process may be long.

Be sure to validate and if necessary correct results before committing anything (specially for records).

Example:

$ python manage.py storage:upgrade-dumps

cli_storage_remove_exported()

Remove files from storage that were used for exporting items

Example:

$ python manage.py storage:remove_exported
$ python manage.py storage:remove_exported --expire-hours=12

create_user_command()

Create a user with given username, password and email.

If user with given username exists it’s noop.

Example:

$ python manage.py users:create -u admin -p admin -e 'admin@example.com' --admin

cli_users_import()

Imports users from JSON or CSV file.

The file is a list of users, where the fields can be:

• username (String), mandatory if there is already a user with given username, it will be skipped.

• email (String), mandatory if there is already a user with given email, it will be skipped.

• password (String)

• first_name (String)

• last_name (String)

• sign_off (String)

• role (String): name of an existing role to assign. The role must exist, otherwise the user will be skipped. Note that the role is case sensitive.

For JSON, it’s a list of object, each user must map field name to value.

For CSV, fields names can be put in the first row of the CSV file. If the first row doesn’t contain fields names, mapping between columns and fields must be done using -f (fields mut be specified in the same order as columns).

File extension is used to detect if the file is JSON or CSV.

Examples:

$ python manage.py users:import path/to/users_list.json

$ python manage.py users:import -f username -f email -f first_name -f last_name path/to/users_list.csv

or, if the first row of your CSV file contains the fields names:

$ python manage.py users:import path/to/users_list.csv

cli_users_copyfromad()

Responsible for importing a user profile from Active Directory (AD) to Mongo.

This command runs on assumption that the user executing this command and the user whose profile need to be imported need not to be the same. Uses ad_username and ad_password to bind to AD and then searches for a user identified by username_to_import and if found imports into Mongo.

Example:

$ python manage.py users:copyfromad --ad_username=ad_uname --ad_password=123 --username_to_import=admin

cli_users_get_auth_token()

Gets auth token.

Generate an authorization token to be able to authenticate against the REST api without starting the client the copy the authorization header.

Example:

$ python manage.py users:get_auth_token --username=admin --password=123123

cli_users_hash_passwords()

Hash all the user passwords which are not hashed yet.

Example:

$ python manage.py users:hash_passwords

cli_generate_vocabularies()

Generate vocabularies.json from flat file

This command generate the vocabularies.json file from a flat text file having the following structure:

Vocabulary Label:
    item 1
    item 2
    item 3

Vocabulary label must end by a colon (:). Indendation is not significant. By default vocabularies maps to an id with the label in lower case where spaces have been replaced by underscode (_). It is possible though to map to an other label/id. For this, you can specify a json file using the --key-map option. It expects a path to json file mapping from vocabulary label set in the flat text file to superdesk id (which will also be used as label). e.g.:

{
    "author types": "author roles",
    "news statuses": null,
    "news urgencies": "urgency",
    "genres": {
        "name": "Genre",
        "extra": {"selection_type": "multi selection"},
    },
    "": {
        "extra": {"type": "unmanageable"}
    }
}

• plain text name will be used as label/id.

• null means that the value must be ignored

• a dict helps to manage more complicated mapping. See below for the keys which can be used.

• empty string (“”) can be used as key when extra must be applied to all non ignored vocabularies

Key which can be used in dictionary:

key	explanation
name	used as label and _id (in lower case and with spaces replaced by underscodes)
extra	a dictionary which will be merged to default one. Useful to add missing keys/values.

You can also specify a path to a file containing a json array using --base which will contain the base vocabularies to use. If not specified, an empty array will be used.

The generated json will be written to vocabularies.json in the current directory.

update_vocabularies_in_items_command()

Update documents in archive and published collections which contain CV related fields: subject, genre, place, anpa_category with corresponding data from vocabularies.

Example:

$ python manage.py vocabularies:update_archive

cli_xml_import()

Import articles into archives.

Example:

$ python manage.py xml:import ninjs data/sample.json

cli_auth_server_register_client()

Register a client to authentication server

A client name is needed, and an id and password will be generated and displayed once the client is registered.

The client will be allowed to access the given scopes using the --scope argument (this argument may be used several times).

Example: Register a new client with name my Superdesk client and allowed to reach items in archive and get users:

$ python manage.py auth_server:register_client "my Superdesk client" -s ARCHIVE_READ -s USERS_READ

cli_auth_server_update_client()

Update an existing client

You need to specify the client ID to update. Specify the parameters that you want to update. You can use --password to set a new password, keep empty to generate a new password.

Examples: Change name of client with id ````:

$ python manage.py auth_server:update_client 0102030405060708090a0b0c –name “some new name”

Regenerate password of client with id 5dad7ee94269dd1d5a78e6a1:

$ python manage.py auth_server:update_client 5dad7ee94269dd1d5a78e6a1 –password

Change scope of client with id 5dad7f064269dd1d5a78e6a2 to allow only ARCHIVE_READ:

$ python manage.py auth_server:update_client 5dad7f064269dd1d5a78e6a2 -s ARCHIVE_READ

cli_auth_server_unregister_client()

Remove a previously registered client Example:

Unregister client with id 0102030405060708090a0b0c:

$ python manage.py auth_server:unregister_client 0102030405060708090a0b0c

cli_auth_server_list_clients()

List clients registered with auth server

The client will be listed with their scopes.

Example:

$ python manage.py auth_server:list_clients

cli_media_fix_links()

Fix media links in content.

When moving an instance to different domain the links to media items can be broken if using Superdesk media storage. This command can check the links and fix them when appropriate.

Usage:

$ python manage.py media:fix_links --prefix "http://example.com/upload"

Use the prefix to define the old url which should be fixed, links starting with other urls won’t be updated.

Options:

-p, --prefix

URL prefix to fix. It will only fix links starting with it.

-r, --resource

Only update specified resource.

-d, --dry-run

Don’t update just print item ids which would be updated.