InterServerSync » History » Revision 20
Revision 19 (Chris Duryee, 10/29/2015 01:25 PM) → Revision 20/34 (Chris Duryee, 10/29/2015 01:26 PM)
h1. InterServerSync
h3. What is ISS?
Inter-Server Sync is a feature designed to help users in two scenarios:
* users who have a connected Foreman/Katello instance and disconnected Foreman/Katello, who want to propogate data from the connected side of their network to the disconnected side.
* users who have a "main" Foreman/Katello and want to propogate some data (but not all data) to other instances. One example is for users who have "blessed" content views that are validated by the IT department, and want to propogate those down elsewhere.
More info is available at https://fedorahosted.org/spacewalk/wiki/InterSpacewalkServerSync.
h3. Goals
Roughly, phase 1 is the minimal "get it working" phase where we have a demoable end-to-end scenario that works via hammer. After that, we will collect feedback and move on to phase 2 which moves some of the grunt work done by hammer to the server, and adds web UI support. Phase 3 enhances the "connected" scenario, and Phase 4 will be for support of additional content types and new types of foreman/katello data.
All code, test, doc and installer changes should be merged by the end of each phase in order to keep PR size down.
h4. Phase 1 "get it working" goals
* allow exporting and importing of products (content will land in Library)
* export/import can occur via hammer, but some filesystem-level access is needed for accessing yum repo exports (remote mount, scp, http, etc)
* only yum is supported
* incremental dumps are supported
Note: phase 1 replaces katello-disconnected script, and solves the same problem that Spacewalk solves with Inter-Spacewalk Sync in a minimal fashion (notably, without a web UI which comes in phase 2 or the disk space usage optimization which comes in phase 3).
h4. Phase 2 "web UI and optimization" goals
* API clean up to allow for bulk export/import with only a few server calls - this is needed for web UI support since we can't rely on hammer
* support for import/export of products and repos via web UI
* entire export is written to disk (both CSVs and repo contents), allowing for a single tarball or iso to have all data.
h4. Phase 3 "do it online and with less disk space" goals
* stream export/import data from one Foreman/katello to another without needing to write a full export to disk
* streaming must be able to be initiated from either the source or destination foreman/katello
* allow export and import without filesystem access to machine (hammer or web ui can provide download of export and upload of import)
h4. Phase 4+ "rinse and repeat" goals
* allow exporting and importing of other data (environments, non-library content views, etc)
* support for ostree, docker, puppet content
h3. Phase 1 Design
There are two areas that need to be updated for phase 1. We need to allow for exporting and importing of yum repo data (including incremental updates), and we need to allow for exporting and importing of product and library CV data. The former is largely provided by Pulp, and the latter is provided by hammer-cli-csv.
We will need to add the Pulp export distributor on newly created yum repos. Additionally, existing repos will need to have the export distributor added in via migration (this work is currently in progress).
Phase 1 allows for hammer-cli-csv doing a lot of "grunt" work in loading the CSV data. This will need to be done server-side in later phases.
h4. Hammer design
Phase 1 uses hammer to do some of the heavy lifting. This gets addressed in phase 2.
<pre>
# export a repository to disk on the Katello server (need to do this for each repo)
hammer repository export --id <id>
# export products to CSV
hammer csv products --csv-export --csv-file products.csv
# replace URL with on-disk location for place to sync from on destination katello
sed -i 's#https://repos.fedorapeople.org/#file:///mnt/imports/#g' /tmp/foo.csv
# import steps
# ensure export is available, then run
hammer csv products --csv-import --csv-file products.csv
# kick off syncs for imported products
hammer repository synchronize --id <id>
</pre>
h4. API modifications
A new API call will be added to perform exports and imports, which will trigger Pulp calls to the export distributor. The export distributor will need to be added to new yum repos, and added via migration to existing repos. We may need to add additional APIs for allowing imports of incremental updates as well.
h4. User Stories
* as a katello hammer user, I would like to be able to export yum repositories to disk with a hammer command. This includes both "full" and "incremental" export. The demoable output would be exporting and showing that there is content that can be synced by another Katello.. On-disk export location can be set in katello.yml
* as a katello hammer user, I would like to be able to export and import product information to a CSV file via hammer, including repository information. The demoable output would be exporting products from Katello A, then importing the product to Katello B. Katellos A and B should have the same products and repos in Library. Note: this story is the one that replaces http://www.katello.org/docs/user_guide/disconnected/index.html with new functionality, and will require a docs update
* as a katello hammer user, I would like to ensure that date-based incremental imports import successfully, using data generated from the product CSV.
h3. Phase 2 Design
The main user-facing output of Phase 2 is the ability to do import/export from the web UI. An additional important feature is that data processing does not occur on the hammer client anymore. This greatly improves resiliency of the import/export process.
h4. API modifications
(need to fill in)
h4. Web UI design/mockup
(need to fill in)
h4. User Stories
* as a hammer user, I would like product CSV creation and processing to occur server-side. Ideally, I will be able to make a call that kicks off a dynflow task to create or import the CSV. The CSV can be uploaded/downloaded, or written to disk in the export directory
* as a hammer user, I would like to run a single command that performs a CSV export and writes any needed repo data to disk
* as a hammer user, I would like to run a single command that performs a CSV import and reads any needed repo data from disk via repo sync. I will be able to override the sync URL (perhaps using something defined in katello.yml) in the call so I don't have to perform a sed statement.
* as a web UI user, I would like to be able to perform the same two import/export actions as above, but without hammer.
h3. Phase 3 Design
This phase is further optimization, focusing on the "connected" scenario where the two foreman/katello instances can talk to each other. Previous phases treated the connected scenario as being solved by the disconnected scenario, which meant that all data was written to disk and optionally transferred via scp or http.
h3. Phase 4 Design
Phase 4 adds additional content types and data to ISS. Detailed design will be added here after phase 2 or phase 3 is complete.