Project

General

Profile

PulpV3GapAnalysis » History » Version 12

Brian Bouterse, 06/11/2018 04:52 PM
Posting Debian content from today's meeting

1 1 Brian Bouterse
h1. PulpV3GapAnalysis
2
3
h1.  Content Tab
4
5
h2.  Content -> Red Hat Repositories
6
7
Katello knows the content URLs from candlepin, matches on the CDN, presents them to the user, the user selects them
8
* Katello creates a Repo tracking this in Pulp with client certificates and CA certificate
9
* Katello specifies custom options from the 'Custom Repo Creation Page' but these use cases are covered in that section
10
11
Katello deletes a Repository
12
13
h2.  Content -> Products
14
15
h3.  Content -> Products -> New Product (used for things like CentOS, SLES, etc)
16
17
All data here is stored only in Katello since this is a Product not a Repository and Pulp doesn't have a concept of a Product
18
Sync Plans will *not* be handled inside of Pulp
19
20
h3.  Content -> Products -> {product_name} -> Repositories
21
22
The user selects a type and content-specific fields are shown.
23
24
h4. Debian:
25
26
h5. Sync Options
27
28
* Upstream URL (str)
29
* Releases (csv list)
30
* Components (csv list)
31
* Architectures (csv list)
32
* Verify SSL (boolean)
33
* Upstream username (str)
34
* Upstream password (str)
35
* Ignore Global http Proxy (bool)
36
37
h5. Publish Options
38
39
* Publish via HTTP (bool)       <----------------------------- PROBLEM AREA
40
41
42
h4. Docker
43
44
* Sync Options
45
46
* Ustream URL (str)
47
48
* Upstream Repository Name (str)
49
50
* Verify SSL (bool)
51
52
* Upstream username (str)
53
54
* Upstream password (str)
55
56
* Ignore Global http Proxy (bool)
57
58
59
h4. File
60
61
h5. Sync Options
62
63
* Upstream URL (str)
64
65
* Verify SSL (boolean)
66
67
* Upstream username (str)
68
69
* Upstream password (str)
70
71
* Ignore Global http Proxy (bool)
72
73
h5. Publish Options
74
75
* Publish via HTTP (bool)       <----------------------------- PROBLEM AREA
76
77
78
h4. OSTree
79
80
h5. Sync Options
81
82
* Upstream URL (str)
83
84
* Upstream Sync Policy (choice):  Latest Only, All History, Custom Depth (with a number specified)    <--- in Pulp2 also specified on distributor
85
86
* Verify SSL (boolean)
87
88
* Upstream username (str)
89
90
* Upstream password (str)
91
92
* Ignore Global http Proxy (bool)
93
94
95
h4. Puppet
96
97
h5. Sync Options
98
99
* Upstream URL (str)
100
101
* Verify SSL (boolean)
102
103
* Upstream username (str)
104
105
* Upstream password (str)
106
107
* Mirror on Sync (boolean)
108
109
* Ignore Global http Proxy (bool)
110
111
h5. Publish Options
112
113
* Publish via HTTP (bool)       <----------------------------- PROBLEM AREA
114
115
116
h4. Yum
117
118
h5. General Fields   <------ not used by Pulp
119
120
* Restrict to Architecture (choice)
121
122
* GPG Key (str)  
123
124
h5. Sync Settings
125
126
* Upstream URL (str)
127
128
* Ignorable Content (multiselect): RPM, DRPM, SRPM, Errata, Distribution
129
130
* Verify SSL (boolean)
131
132
* Upstream username (str)
133
134
* Upstream password (str)
135
136
* Download Policy (choice): (On Demand, Background, Immediate)   <---- Background does not have a strong use case
137
138
* Mirror on Sync (bool)
139
140
* Ignore Global http Proxy (bool)
141
142
* SSL CA Cert (str)
143
144
* SSL Client Cert (str)
145
146
* SSL Client Key(str)
147
148
h5. Publish Settings
149
150
* Checksum: (choice) Default, sha256, sha1   <----- for all repodata including primary.xml
151
152
153 3 Brian Bouterse
h3.  Content -> Products -> {product_name} -> Repositories -> {repository_name}
154
155
This displays a created repository.
156
157
Katello allows the user to upload a package
158
159
* Receives the data from the user, sends it to Pulp
160
161 4 Brian Bouterse
* Relies on Pulp to fully parse the metadata and create the unit                <------- REQUIREMENT: must have Pulp determine all metadata
162 3 Brian Bouterse
163
* Associates the the unit with the repository
164
165
Katello Reads a content Summary on this page
166
167 4 Brian Bouterse
h5.  Content -> Products -> {product_name} -> Repositories -> {repository_name} -> Select Action -> Sync Now
168 1 Brian Bouterse
169 4 Brian Bouterse
Katello tells the remote associated with the repository to sync
170 3 Brian Bouterse
171 4 Brian Bouterse
h5.  Content -> Products -> {product_name} -> Repositories -> {repository_name} -> Select Action -> Advaced Sync
172 1 Brian Bouterse
173
Katello can peroform an 'Advnaced Sync':
174 3 Brian Bouterse
175 4 Brian Bouterse
Optimized Sync - Normal sync, presented
176 3 Brian Bouterse
177 4 Brian Bouterse
Complete Sync - force-full on sync and force-full on publish      <--------------------- GAP because we don't have force-full
178 1 Brian Bouterse
179 4 Brian Bouterse
Validate Content Sync - performs a checksum validation on all packages
180 1 Brian Bouterse
181 4 Brian Bouterse
* True Purpose:  Validate existing downloaded content and redownload if the file(s) are missing or corrupt, redownload them.    <-------- GAP
182 1 Brian Bouterse
183 4 Brian Bouterse
184
h5.  Content -> Products -> {product_name} -> Repositories -> {repository_name} -> Select Action -> Republish Repository Metadata
185
186
Republishes the metadata.
187
188
* Katello would create a new Publication and update the Distribution
189
190
191
h5.  Content -> Products -> {product_name} -> Repositories -> {repository_name} -> Select Action -> Delete a Repository
192
193
Deletes a repository
194
195
196 3 Brian Bouterse
h3.  Content -> Products -> {product_name} -> Repositories
197
198
This is the index view of all repositories
199
200
Repsitories in Katello can have the same name, but Pulp enforces a unique name on repositories globally      <--------- GAP
201
202
Katello takes a Product ID which resolves to a set of repos. Katello fetches this set of repos. For each repo we need to fetch:
203
204
* name (str)
205
206
* type (str), e.g. 'yum'
207
208
* sync status, e.g. 'Not synced, Pending, Error'     <-------------------------  GAP this would require a second call to load the data per Remote
209
210
* Content Summary, e.g. 2 packages, 5 errata, etc. Similarly for other types.
211
212
213
Katello can trigger a sync of one or more Repositories at once.
214
215
* Trigger the sync on one or more Remotes as independant calls
216
217
218
Katello can trigger a delete of one or more Repositories at once.
219
220
* Trigger the delete call to Pulp as independant calls
221
222
223
Search/Filtering of the list of Repositories, for Repository attributes
224
225
* content_type: the type of content
226
227
* content_view_id: the id of the content View              <-------- not in Pulp anywhere currently
228
229
* ignore_global_proxy                      <--------- GAP area, not currently in Pulp, but probably should be
230
231
* name
232
233
* product
234
235
* redhat          <----------   Anything added from Red Hat "Products" page in Katello gets Red Hat.
236
237
238
Search/Filtering of the list of Repositories, for content units
239
240
* distribution_arch: 
241
242
* distribution_bootable       <-----------    if Katello can detect if it has a vmlinuz init.rd it knows the distribution is bootable. Detected at the end of every sync.
243 1 Brian Bouterse
244 3 Brian Bouterse
* distribution_family
245
246
* distribution_uuid
247
248 1 Brian Bouterse
* distribution_variant
249
250 4 Brian Bouterse
* distribution_version
251 1 Brian Bouterse
252
253
*NOTE: Must not have to make a call for each item in a list page. Must be able to make one call.*
254
255
256 4 Brian Bouterse
h3.  Content -> Products -> {product_name} -> Repositories -> {repository_name} -> Packages
257 1 Brian Bouterse
258 4 Brian Bouterse
Lists packages in a repository (the latest repository version)
259 1 Brian Bouterse
260 4 Brian Bouterse
Removing packages from the repository
261 1 Brian Bouterse
262 4 Brian Bouterse
* Can remove n packages from the repository
263
* Republish, Redistribute the repository
264
265
266
h2. Content -> Content Credentials
267
268
h3. Content -> Content Credentials -> GPG Keys
269
270
GPG keys can be created and stored by Katello
271
Pulp3 recommendation is to use pulp_file to hold the GPG keys hosted for clients to receive
272
273
274
h3. Content -> Content Credentials -> SSL Certificate  (GAP. This whole section is a GAP b/c Pulp doesn't "host" SSL certs, you have to manually install them on the filesystem first)
275
276
Stores SSL certificates for use by Pulp at sync time as CA cert, client cert, or client key
277
278
* name
279
* value
280
281
Supports updating them
282
Support deleting them
283
Support searching them (name, organization_id)
284
SSL Certs are per-product, so Katello needs some way to restrict the set of available SSL certs for the current "product"
285
286
287
h2. Content -> Sync Plans
288
289
Sync plans will not be handled by Pulp 3, Katello/Foreman will handle scheduling.
290
291
292
h2. Content -> Sync Status
293
294
Show the most-recent sync status from dynflow data. That data is populated by task status results from Pulp, which needs to contain at a minimum:
295
296
* start time
297
* create time
298
* end time
299
* state
300
* progress reports
301
* fatal errors
302
* non-fatal errors
303 1 Brian Bouterse
304
h2.  Content -> Lifecycle Environments
305 5 Brian Bouterse
306
Creates a lifecycle environment
307
308
* Does *not* involve Pulp
309
310
311
h3. Content -> Lifecycle Environments -> {name} -> Details
312
313
Each lifecycle environment has a 'Registry Name Pattern'.      <------- GAP (specific to Docker only)
314
315
* Likely going to be on the Distributor
316
317
* Katello would use the template to produce a concrete value to set on the Distributor
318
319
* Important to ensure that two Distribution don't both receive the same concrete values
320
321
322
h3. Content -> Lifecycle Environments -> {name} -> Content Views
323
324
Filterable by:
325
326
* composite
327
* label
328
* name
329
* organization_id
330
331
332
h3. Content -> Lifecycle Environments -> {name} -> Yum Repositories
333
334
Content will come from CV section on Yum Repositories
335
336
337
h3. Content -> Lifecycle Environments -> {name} -> Errata
338
339
Content will come from CV section on Errata
340
341
342
h3. Content -> Lifecycle Environments -> {name} -> Packages
343
344
Content will come from CV section on Packages
345
346
347
h3. Content -> Lifecycle Environments -> {name} -> Puppet Modules
348
349
Content will come from CV section on Puppet Modules
350
351
352
h3. Content -> Lifecycle Environments -> {name} -> Container Image Tags
353
354
Content will come from CV section on Container Image Tags
355
356
357
h3. Content -> Lifecycle Environments -> {name} -> OSTree Branches
358
359
Content will come from CV section on OSTree Branches
360
361 6 Brian Bouterse
h2.  Content -> Content Views
362 1 Brian Bouterse
363 6 Brian Bouterse
h3. Content -> Content Views -> {name} -> Yum Repositories
364 1 Brian Bouterse
365 6 Brian Bouterse
List/Remove/Add one or more repositories to the Content View
366
* Does *not* involve Pulp
367 1 Brian Bouterse
368 6 Brian Bouterse
h3. Content -> Content Views -> {name} -> Yum Filters
369
370 9 Brian Bouterse
Katello filters combine together (whitelist/blacklist/etc), and can be heavily modified by users to ultimately produce a set of packages.    <------GAP:  Katello would have to store huge lists of packages/errata to maintain this design.
371
372
373
h5. Package Filters
374
375
* Select RPMs using include or exclude filters to be included/disincluded from the content view.
376
* package name. Also supports wildcard. - an attribute of the metadata
377
* architecture. An attribute of the metadata
378
* version, lt, gt, range, etc. An attribute of the metadata
379
380
Checkbox with 'include all RPMs with no errata'. Solves a practical issue whereby packages that received no errata are not included in a content view when the user applies a filter that only includes packages referenced as errata.
381
382
383
h5. Package Group Filter
384
385
Select package groups to include or remove rpms
386
* name - an attribute of the metadata
387
* product - the katello stored attribute
388
* repository - the repo containing that unit
389
* description - an attribute of the metadata
390
391
392
h5. Errata by ID Filter                                                                                 <------ GAP: must be able to ask Pulp filter info and exclude 
393
394
Filters to produce a list and then you can select from the list.
395
396
filterable on errata attributes
397
* type (multiselect) i.e. security, enhancement, bugfix
398
* date either or choice:  i.e. updated on, Issued on w/ start/end date
399
* bug
400
* cve
401
* id
402
* issued
403
* package
404
* package_name
405
* reboot_suggested
406
* severity
407
* title
408
* type
409
* updated
410
411
412
h5. Errata by Date
413
414
Filters to produce a list. You *cannot* select from the list.
415
416
filterable on errata attributes
417
* type (multiselect) i.e. security, enhancement, bugfix
418
* date either or choice:  i.e. updated on, Issued on w/ start/end date
419
420
421
h3. Content -> Content Views -> {name} -> Apt Repositories
422
423
No filtering. Add/remove Debian repositories from the content view.
424
425
426
h3. Content -> Content Views -> {name} ->  File Repositories
427
428
No filtering. Add/remove file repositories from the content view.
429
430
431
h3. Content -> Content Views -> {name} -> Puppet Modules
432
433
Each module can only be included once. Can't have 2+ versions of the same module in one content view.
434
435
Attributes:
436
437
* Name
438
* Author
439
* Version
440
441
442
h3. Content -> Content Views -> {name} ->  Container Images
443
444
Filtering is 'tag' based and used to produce a concrete set of image names.
445
446
447
h3. Content -> Content Views -> {name} -> OSTree Content
448
449
No filtering. Add/remove ostree repositories from the content view.
450
451
452
h3. Content -> Content Views -> {name} -> History
453
454
Not related to Pulp.
455
456
457
h3. Content -> Content Views -> {name} -> Tasks
458
459
Not related to Pulp.
460
461
462
h2. Content -> Content Views -> {name} -> Publish
463
464
The concrete content set from all filters is computed and those units are associated with the CV repositories.
465
Those content view repositories are then published via the Distributions that host Library.
466
467
468
h5. Promotion
469
470
Other "promotion" events will cause existing Publications to be exposed via existing/new Distributions associated with the lifecycle environment.
471
472
When delivering content to a capsule the "Force Yum Metadata Regernation" is used to cause Katello to inspect the published times of the repo on the main satellite server and the capsule. If nothing changed then don't "resync" the capsule's repo.
473
474
475
h5. Regenerate Repository Metadata
476
477
Causes Pulp to force-full publish.
478
479
480
h5. Incremental Update
481
482
Take an existing Content View and add/remove packages and errata w/ dependency resolution from the content set. Say a V1 exists, this would create a V1.1.
483
484
485
h5. Remove
486
487
Un-distributes and potentially delete the publications for one or more repos backing the content views.
488
489 1 Brian Bouterse
h2.  Content -> Deb Packages
490
491 12 Brian Bouterse
List:
492
493
* Name
494
* Version
495
* Architecture
496
497
Filter options:
498
    
499
* architecture
500
* checksum
501
* filename
502
* name
503
* version
504
505
506
h5. Details tab
507
508
Details of the Debian package
509
510
511
h5. Repositories tab
512
513
List the Debian repositories containing the package
514
515
Filtering by:
516
517
* Lifecycle Environment
518
* Organization
519 1 Brian Bouterse
520
h2.  Content -> Container Image Tags
521
522
523
h2.  Content -> Errata
524
525
526
h2.  Content -> Files
527
528
529
h2.  Content -> OSTree Branches
530
531
532
h2.  Content -> Packages
533
534
535
h2.  Content -> Puppet Modules
536
537
538
539
h1.  Hosts -> Content Hosts
540
541
542
h1.  Non UI things
543
544
* the API endpoint that clients upload their enabled repos
545
* the API endpoint that clients upload their package profiles
546
* the API endpoint that clients register
547
* the API endpoint that clients unregister
548
* speed throttling and other global settings?
549 2 Justin Sherrill
* Errata mailer
550
* smart proxy page/details
551 1 Brian Bouterse
552
553
554
h1.  Terminology
555
556
Candlepin Manifest - Defines Products, Subscriptions, and a Content Sets
557
Product - A collection of repositories. A repository can only belong to one product
558
Repository Set - Has a name, Label, and URL of the form:  /content/rhel/server/7/$RELVER/$BASEARCH/os/