Superdesk CLI

In superdesk/server folder you can find manage.py file which you can use to run commands from CLI:

$ python manage.py

With no other argument it will output list of all available commands. To run specific command you use its name and any params needed:

$ python manage.py users:create -u admin -p admin

Creating new command

You can create new commands extending superdesk.Command and registering it using superdesk.command() call:

import superdesk

class HelloWorldCommand(superdesk.Command):

    option_list = [
        superdesk.Option('--name', '-n', required=True),
    ]

    def run(self, name):
        print('hello {}'.format(name))


superdesk.command('hello:word', MyCommand())

We use Flask-Script under the hood so you can get more info there.

Superdesk commands

app:clean_images

class CleanImages

This command will remove all the images from the system which are not referenced by content.

It checks the media type and calls the correspoinding function as s3 and mongo requires different approaches for handling multiple files. Probably running db.repairDatabase() is needed in Mongo to shring the DB size.

Example:

$ python manage.py app:clean_images

app:deleteArchivedDocument

class DeleteArchivedDocumentCommand

Deletes a Text Archive document(s) from both Mongodb and ElasticSearch.

It deletes digital package and the story given by id. It accepts one or more ids separated by space.

Example:

$ manage.py app:deleteArchivedDocument 588c1df11d41c80928015601 588c1b901d41c805dce70df0

app:index_from_mongo

class IndexFromMongo

Index the specified mongo collection in the specified elastic collection/type.

This will use the default APP mongo DB to read the data and the default Elastic APP index.

Use -f all to index all collections.

Example:

$ python manage.py app:index_from_mongo --from=archive
$ python manage.py app:index_from_mongo --all

app:initialize_data

app:flush_elastic_index

class FlushElasticIndex

Flush elastic index.

It removes elastic index, creates a new one and index it from mongo. You must specify at least one elastic index to flush: --sd (superdesk) or --capi (content api)

Example:

$ python manage.py app:flush_elastic_index --sd
$ python manage.py app:flush_elastic_index --capi
$ python manage.py app:flush_elastic_index --sd --capi

app:prepopulate

app:populate

app:rebuild_elastic_index

class RebuildElasticIndex

Rebuild the elastic indexes from existing data.

It creates new index with same alias as the configured index, puts the new mapping and removes the old index.

Example:

$ python manage.py app:rebuild_elastic_index
$ python manage.py app:rebuild_elastic_index --resource=items
$ python manage.py app:rebuild_elastic_index --resource=archive

app:run_macro

class RunMacro

Executes a macro by given name and optional keyword arguments.

Example:

$ app:run_macro --name clean_keywords --kwargs {"repo":"archived"}

app:scaffold_data

app:updateArchivedDocument

class UpdateArchivedDocumentCommand

Update metadata for a document in both Mongodb and ElasticSearch by supplying ids field to update and the value for the update

--field

Field name

--value

Value to be set in the field

--parseNeeded

Optional. True if value is a complex type, not an int or string

Example:

$ manage.py app:updateArchivedDocument --ids='["588c1df11d41c80928015601","588c1b901d41c805dce70df0"]'
    --field=anpa_category
    --value=[{"scheme" : null,"qcode" : "f","subject" : "04000000","name" : "Finance"}]
    --parseNeeded=True

archive:remove_expired

audit:purge

class PurgeAudit

Purge audit collection.

Entries are purged if the related item is no longer in ‘Production’. Other entries are deleted after the configured time period.

Example:

$ python manage.py audit:purge
$ python manage.py audit:purge --expiry_minutes=20

content_api:remove_expired

class RemoveExpiredItems

Remove expired items from the content_api items collection.

By default no items expire there, you can change it using CONTENT_API_EXPIRY_DAYS config.

Example:

$ python manage.py content_api:remove_expired

data:generate_update

class GenerateUpdate

Generate a file where to define a new data update.

Example:

$ python manage.py data:generate_update --resource=archive

data:upgrade

class Upgrade

Runs all the new data updates available.

If data_update_id is given, runs new data updates until the given one.

Example:

$ python manage.py data:upgrade

data:downgrade

class Downgrade

Runs the latest data update backward.

If data_update_id is given, runs all the data updates backward until the given one.

Example:

$ python manage.py data:downgrade

ingest:clean_expired

ingest:provider

class AddProvider

Add ingest provider.

Example:

$ python manage.py ingest:provider --provider='{"update_schedule" : { "minutes" : 5, "seconds" : 0 },
    "idle_time" : { "hours" : 0, "minutes" : 0 }, "content_expiry" : 2880, "name" : "aap-demo",
     "source" : "aap-demo", "feeding_service" : "rss",
     "config" : { "url" : "https://abcnews.go.com/abcnews/primetimeheadlines", "field_aliases" : [ ] },
     "feed_parser" : null, "content_types" : [ "text" ]}'

ingest:update

publish:enqueue

publish:transmit

class PublishContent

Deliver items from publish_queue to destinations.

Example:

$ python manage.py publish:transmit

session:gc

class RemoveExpiredSessions

Remove expired sessions from db.

Using SESSION_EXPIRY_MINUTES config.

Example:

$ python manage.py session:gc

schema:migrate

class SchemaMigrateCommand

Migrate elastic schema if needed, should be triggered on every deploy.

It compares version set in code (latest) to one stored in db and only updates schema if those are different.

Current version is read from settings and fallbacks to superdesk.SCHEMA_VERSION, so that you can avoid migration via settings file if needed.

Example:

$ python manage.py schema:migrate

Storage Dumps and Records

Note

You can check Database Recording for details on how to use following storage:… commands.

class StorageDump(func=None)

Dump collections from MongoDB

Note: this command should only be used for development purpose. Use MongoDB official commands to dump or restore databases in production.

Dump whole collection into either separate JSON files (one per collection) or a single one (with a single root object).

Single file are easier to copy, while separate JSON files are easier to check.

A dump is full Superdesk save, as opposed to a record which only store change made in a database (more like a diff).

Example:

Do a full database dump with a name and description to a single file:

$ python manage.py storage:dump -n "demo-instance" -D "this dump includes some test data (desks, users) to run a
basic demo of Superdesk" --single
class StorageRestore(func=None)

Restore MongoDB collections dumped with storage:dump

Example:

$ storage:restore foobar_superdesk_dump
class StorageStartRecording(func=None)

Record changes made in database until the command is stopped

This command is intended for developers to help producing specific state (e.g. for tests), or to create a specific Superdesk instance e.g. for a demo.

If you specify a base dump (with --base-dump), the database will be restored to this dump before starting the record, and it will be associated with the record.

If no base dump is specified, the database should be in the same state as when the record has been done when you restore it.

If you use the --full-document option, the whole document will be stored in the record in case of update (instead of just diff), this will result in a bigger dump file, but may be applied to a database even if it was not exactly in the same state as when the record has been done.

You may want to use --collection to record change only on the collections you’re interested in, and avoir side effects.

Example:

Record change in vocabularies only, with a name and description, and base on “base_test_e2e_dump” dump:

$ python manage.py storage:record -b "base_test_e2e_dump" -c vocabularies -n "test_categories"-D "prepare
  instance for categories end-to-end tests"
class StorageRestoreRecord(func=None)

Restore Superdesk record

This command is to be used with a record dump, not a full database archive.

Example:

$ storage:restore-record record-for-some-e2e-test
class StorageList(func=None)

List Superdesk Dumps and Records

class StorageMigrateDumps(func=None)

Apply migration scripts on all dumps and records

Note: the backend MUST NOT be running while this command is used.

Migration scripts are first applied on each records: changes on collections used in the record are merged to the record.

Then for full dump, migration are normally applied and dumps are updated.

The whole process may be long.

Be sure to validate and if necessary correct results before committing anything (specially for records).

Example:

$ python manage.py storage:upgrade-dumps
class RemoveExportedFiles

Remove files from storage that were used for exporting items

Example:

$ python manage.py storage:remove_exported
$ python manage.py storage:remove_exported --expire-hours=12
class CreateUserCommand

Create a user with given username, password and email.

If user with given username exists it’s noop.

Example:

$ python manage.py users:create -u admin -p admin -e 'admin@example.com' --admin
class ImportUsersCommand

Imports users from JSON or CSV file.

The file is a list of users, where the fields can be:

  • username (String), mandatory if there is already a user with given username, it will be skipped.

  • email (String), mandatory if there is already a user with given email, it will be skipped.

  • password (String)

  • first_name (String)

  • last_name (String)

  • sign_off (String)

  • role (String): name of an existing role to assign. The role must exist, otherwise the user will be skipped. Note that the role is case sensitive.

For JSON, it’s a list of object, each user must map field name to value.

For CSV, fields names can be put in the first row of the CSV file. If the first row doesn’t contain fields names, mapping between columns and fields must be done using -f (fields mut be specified in the same order as columns).

File extension is used to detect if the file is JSON or CSV.

Examples:

$ python manage.py users:import path/to/users_list.json

$ python manage.py users:import -f username -f email -f first_name -f last_name path/to/users_list.csv

or, if the first row of your CSV file contains the fields names:

$ python manage.py users:import path/to/users_list.csv
class ImportUserProfileFromADCommand

Responsible for importing a user profile from Active Directory (AD) to Mongo.

This command runs on assumption that the user executing this command and the user whose profile need to be imported need not to be the same. Uses ad_username and ad_password to bind to AD and then searches for a user identified by username_to_import and if found imports into Mongo.

Example:

$ python manage.py users:copyfromad --ad_username=ad_uname --ad_password=123 --username_to_import=admin
class GetAuthTokenCommand

Gets auth token.

Generate an authorization token to be able to authenticate against the REST api without starting the client the copy the authorization header.

Example:

$ python manage.py users:get_auth_token --username=admin --password=123123
class HashUserPasswordsCommand

Hash all the user passwords which are not hashed yet.

Example:

$ python manage.py users:hash_passwords
class GenerateVocabularies

Generate vocabularies.json from flat file

This command generate the vocabularies.json file from a flat text file having the following structure:

Vocabulary Label:
    item 1
    item 2
    item 3

Vocabulary label must end by a colon (:). Indendation is not significant. By default vocabularies maps to an id with the label in lower case where spaces have been replaced by underscode (_). It is possible though to map to an other label/id. For this, you can specify a json file using the --key-map option. It expects a path to json file mapping from vocabulary label set in the flat text file to superdesk id (which will also be used as label). e.g.:

{
    "author types": "author roles",
    "news statuses": null,
    "news urgencies": "urgency",
    "genres": {
        "name": "Genre",
        "extra": {"selection_type": "multi selection"},
    },
    "": {
        "extra": {"type": "unmanageable"}
    }
}
  • plain text name will be used as label/id.

  • null means that the value must be ignored

  • a dict helps to manage more complicated mapping. See below for the keys which can be used.

  • empty string (“”) can be used as key when extra must be applied to all non ignored vocabularies

Key which can be used in dictionary:

key

explanation

name

used as label and _id (in lower case and with spaces replaced by underscodes)

extra

a dictionary which will be merged to default one. Useful to add missing keys/values.

You can also specify a path to a file containing a json array using --base which will contain the base vocabularies to use. If not specified, an empty array will be used.

The generated json will be written to vocabularies.json in the current directory.

class UpdateVocabulariesInItemsCommand

Update documents in archive and published collections which contain CV related fields: subject, genre, place, anpa_category with corresponding data from vocabularies.

Example:

$ python manage.py vocabularies:update_archive
class ImportCommand

Import articles into archives.

Example:

$ python manage.py xml:import ninjs data/sample.json
class RegisterClient

Register a client to authentication server

A client name is needed, and an id and password will be generated and displayed once the client is registered.

The client will be allowed to access the given scopes using the --scope argument (this argument may be used several times).

Example: Register a new client with name my Superdesk client and allowed to reach items in archive and get users:

$ python manage.py auth_server:register_client "my Superdesk client" -s ARCHIVE_READ -s USERS_READ
class UpdateClient

Update an existing client

You need to specify the client ID to update. Specify the parameters that you want to update. You can use --password to set a new password, keep empty to generate a new password.

Examples: Change name of client with id ````:

$ python manage.py auth_server:update_client 0102030405060708090a0b0c –name “some new name”

Regenerate password of client with id 5dad7ee94269dd1d5a78e6a1:

$ python manage.py auth_server:update_client 5dad7ee94269dd1d5a78e6a1 –password

Change scope of client with id 5dad7f064269dd1d5a78e6a2 to allow only ARCHIVE_READ:

$ python manage.py auth_server:update_client 5dad7f064269dd1d5a78e6a2 -s ARCHIVE_READ
class UnregisterClient

Remove a previously registered client Example:

Unregister client with id 0102030405060708090a0b0c:

$ python manage.py auth_server:unregister_client 0102030405060708090a0b0c
class ListClients

List clients registered with auth server

The client will be listed with their scopes.

Example:

$ python manage.py auth_server:list_clients