Explain the tier-naming axis: `Manual` means external-origin, not hand-entered

[dimitri-yatsenko]

The problem

The two most-used tier names describe opposite axes, and each one names the axis the other actually means:

• Manual names the writer ("a person"), but the tier means the origin (data from outside the pipeline).
• Imported names the origin ("from elsewhere"), but the tier means the writer (the table's own make()).

The docs are consistent and correct about what the tiers mean. It is the class names that cross over, and a reader who reasons from the names alone will land in the wrong place.

The docs already explain Manual in different words than its name

• Table Declaration §1.2 — Manual is "Data inserted at runtime from outside the pipeline (users, instruments, ingestion scripts)". Two of the three examples are automated.
• Computation Model — "Manual and Lookup tables are not auto-populated; they are the entry points where a pipeline's data originates."
• Fan-Out Ingestion — "inserts into several entry-point tables (Manual or Imported)".

So the concept is already named "entry point" throughout the prose. Only the class is called Manual.

The failure mode this produces

A designer building an automated feed reasons, reasonably: "this table isn't filled manually — a sync script fills it — so it isn't Manual. It comes from outside, so it's Imported." They declare dj.Imported.

But Imported is defined as "populated by make()", and this table has no make() — it's filled by insert from elsewhere. So the insert is refused, and the fix that presents itself is allow_direct_insert=True rather than a corrected tier.

The result is a recognizable and diagnosable shape:

• an Imported table with no make(), which reads to every later reader as a table whose make() was forgotten;
• populate() on it silently doing nothing rather than erroring;
• a permanent allow_direct_insert=True suppressing a guard that was reporting a real problem;
• the pipeline's actual boundary with the outside world drawn one table away from where it is.

Every step of that reasoning is sound except the premise that "manual" means "by hand."

Suggested docs changes (the near-term ask)

1. State the axis explicitly, once, where tiers are introduced: the tier says how rows arrive, not who causes them to arrive. Rows inserted from outside the table → Manual; rows produced by the table's own make() → Imported (reads an external source) or Computed (derives from other tables).
2. Say plainly that "manual" means external-origin, not hand-entered — an automated ingestion script inserts into a Manual table. The reference table hints at this with "ingestion scripts"; it deserves a sentence, not a parenthetical.
3. Add the failure mode as a short "common mistake" callout — an Imported table with no make(), and allow_direct_insert as its tell. It is cheap to check and hard to spot by eye.
4. Consider a small 2×2 (origin: external / derived × writer: inserted / make()) — it makes Lookup and Part sit naturally alongside the other three and shows why only four of the cells are populated.

Forward-looking: a better name

dj.Entry would name the thing itself — a table where data enters the pipeline, by whatever hand — and it matches the vocabulary the explanation pages already use. It carries no implication about who or what does the writing, which is the whole source of the confusion.

That is a datajoint-python change, not a docs change, so it needs a decision there. Filing it here because the explanation is the fix available now and the naming question follows from it. If it were taken up, the obvious shape is dj.Entry as the primary name with dj.Manual kept permanently as a backward-compatible alias — no deprecation, no migration, existing pipelines and tutorials keep working, and new material can teach the clearer name. Renaming Imported is a harder call and probably not worth it; once the axis is stated, Imported reads correctly as "imported by this table."

Happy to open a PR for items 1–3 if the framing looks right.

[dimitri-yatsenko]

Revising the last section. I argued above that renaming Imported probably isn't worth it and that it "reads correctly once the axis is stated." Having gone through how the docs actually use the words, I no longer think that holds. Lookup / Entry / Ingest / Computed is the better set, with one prerequisite.

What changed my mind

1. The docs already name an Imported table Ingest. The canonical fan-out example is literally:

class Ingest(dj.Imported):

When the explanation needed a name for this kind of table, it reached for "Ingest" — not "Import".

2. Imported is passive, and that is the whole bug. A passive participle describes the data ("this data was imported"), which puts the name on the origin axis. Ingest describes the table's action, which is the writer axis — the one that actually distinguishes the tier. So renaming both puts all four names on a single axis, answering one question: what puts rows in this table?

Tier	What puts rows in it
Lookup	the code, via contents
Entry	a writer outside the table — a person, an instrument, a feed
Ingest	the table's own make(), reading an external source
Computed	the table's own make(), deriving from other tables

Entry and Ingest also pair naturally: data is entered into one; the other ingests it. Pushed vs pulled.

3. The docs are already writing disclaimers against the current name. From the basics tutorial:

"Manual tables are not necessarily populated by hand—they contain data entered into the pipeline by operators, instruments, or ingestion scripts…"

A name that needs a corrective sentence every time it is taught is not doing its job. That single line is the strongest argument in this issue.

The prerequisite, and it is not optional

"Ingestion" is currently used in the docs for both tiers, which is a real ambiguity independent of any rename:

• pointing at Imported — "an ingestion routine — typically an Imported table's make()", "the ingesting table", "one ingestion step", the fan-out ingestion pattern itself;
• pointing at Manual — "inserted from outside the pipeline (forms, instruments, ingestion scripts)", in five places: reference/specs/table-declaration.md:26, how-to/define-tables.md:33, tutorials/basics/02-schema-design.ipynb (×3), tutorials/advanced/sql-comparison.ipynb:2070.

Shipping dj.Ingest while five places define the other class as the one written by "ingestion scripts" would point the word at the wrong tier and make today's confusion worse, not better — a designer with an ETL script would read "ingestion" and pick dj.Ingest, which is exactly the error we are trying to prevent.

So the rename has a hard dependency: first reword those five to "automated feeds" or "loader scripts", reserving "ingest" for the tier that fetches for itself. That cleanup is worth doing on its own merits even if no class is ever renamed — it is the same overload, just currently invisible.

Revised proposal

1. Docs items 1–4 above — the axis, stated once, plus the common-mistake callout. Independent of any rename.
2. Purge the "ingestion scripts" overload from the five Manual definitions.
3. Then, in datajoint-python: dj.Entry and dj.Ingest as primary names, with dj.Manual and dj.Imported kept permanently as backward-compatible aliases — no deprecation, no migration, no broken tutorials.

Computed and Lookup stay as they are; both already sit on the right axis.

[dimitri-yatsenko]

Converging on a proposal. Two refinements to the above, the second of which retires the AutoIngest idea I was weighing.

"Entry scripts"

The collision I raised — "ingestion" naming both tiers — is best fixed by giving the pushing-in activity its own name that follows the class: an entry script writes to an Entry table. That frees "ingest" to mean the tier that fetches for itself, and it fixes the ambiguity at the root rather than working around it.

This replaces "ingestion scripts" at the five sites listed above (reference/specs/table-declaration.md:26, how-to/define-tables.md:33, tutorials/basics/02-schema-design.ipynb ×3, tutorials/advanced/sql-comparison.ipynb:2070).

It also settles a question I had been going back and forth on: whether the tier should be AutoIngest rather than Ingest, with the prefix disambiguating against the industry's use of "ingestion" for ETL. That prefix was a crutch for a vocabulary problem — and "entry script" solves the vocabulary problem directly. Plain Ingest. The prefix would also have introduced a false asymmetry, since Computed is equally auto-populated and would carry no prefix.

The quad: Lookup / Entry / Ingest / Compute

All four sit on one axis — what puts rows in this table — and the grammar itself teaches the axis:

Tier	Part of speech	What puts rows in it
Lookup	noun	the code, via contents
Entry	noun	something outside the table — a person, an instrument, an entry script
Ingest	verb	the table itself, via make(), reading an external source
Compute	verb	the table itself, via make(), deriving from other tables

The nouns are passive tables; the verbs are what DataJoint automates. Ingest and Compute are precisely the two things populate() does — so the split is not a stylistic choice, it is the product boundary written into the names. A reader who never reaches the tier table can still infer which tables they fill and which the framework fills, from the shape of the words alone.

That is exactly what the current set cannot do. Manual / Imported / Computed are three participles spanning both sides of that boundary, and two of them name the wrong axis: Manual names the writer but means the origin; Imported names the origin but means the writer.

Renaming Computed → Compute is the same correction as Imported → Ingest, applied to the other automated tier: a participle describes the data ("this data was computed"), a verb describes the table's work. Doing one and not the other would leave the active pair looking unlike each other.

The one risk worth naming: "compute" as a mass noun means computing capacity. In class TuningCurve(dj.Compute) the verb reading dominates and DataJoint has no competing use of the word, so I think it is small — but it is the part of this proposal I would most like a second opinion on. The conservative fallback is Lookup / Entry / Ingest / Computed: one fewer rename, at the cost of the symmetry.

Sequencing

The docs work does not depend on any rename and is worth doing on its own:

1. State the axis once, where tiers are introduced — what puts rows in the table.
2. Adopt entry script; retire "ingestion scripts" at the five sites.
3. Add the common-mistake callout: an auto-populated table with no make(), and allow_direct_insert as its tell.

Then, as a datajoint-python decision: dj.Lookup / dj.Entry / dj.Ingest / dj.Compute as the primary names, with dj.Manual and dj.Imported (and dj.Computed) kept permanently as backward-compatible aliases — no deprecation, no migration, no broken tutorials or third-party code. New material teaches the clearer set.

I can open a PR for the three docs items on this framing.

[dimitri-yatsenko]

Proposal

Rename the tiers so that all four sit on one axis — what puts rows in this table — and let the grammar carry the distinction:

Tier		What puts rows in it
Lookup	noun	the code, via contents
Entry	noun	something outside the table — a person, an instrument, an entry script
Ingest	verb	the table itself, via make(), reading an external source
Compute	verb	the table itself, via make(), deriving from other tables

The nouns are passive tables; the verbs are what DataJoint automates. Ingest and Compute are precisely the two things populate() does, so the split is not stylistic — it is the product boundary written into the names. A reader who never reaches the tier table can still tell which tables they fill and which the framework fills, from the shape of the words.

The present set cannot do that. Manual, Imported and Computed are three participles spanning both sides of that boundary, and two of them name the wrong axis: Manual names the writer but means the origin, Imported names the origin but means the writer. A participle describes the data — "this data was imported" — which is why both automated tiers read as descriptions of their contents rather than of their work.

Entry script is the companion term: the thing that writes to an Entry table. It replaces "ingestion scripts" at the five sites where that phrase currently describes Manual — reference/specs/table-declaration.md:26, how-to/define-tables.md:33, tutorials/basics/02-schema-design.ipynb (×3), tutorials/advanced/sql-comparison.ipynb:2070 — which today point the word "ingestion" at the tier that does not ingest.

Rollout

The old names have been in constant use for a decade, so they are never retired — no deprecation, no migration, no broken tutorials, no broken third-party code.

• 2.3.3 — introduce dj.Entry, dj.Ingest and dj.Compute as aliases. Nothing changes for anyone; the new names simply become available.
• 2.4 — switch the documentation to the new names: the tier tables, the explanation pages, the tutorials, and the diagram legend. dj.Manual, dj.Imported and dj.Computed remain fully supported and continue to work everywhere.

Existing pipelines never need to change. New material teaches the clearer set.

Docs work that does not wait on the rename

Worth doing on its own merits, in either vocabulary:

1. State the axis once, where tiers are introduced — what puts rows in the table, not who causes it.
2. Say plainly that data arriving from an automated feed still goes to a written-from-outside table; the tier is about how rows arrive, not whether a human typed them.
3. Add a common-mistake callout: an auto-populated table with no make(), filled by insert from elsewhere, with allow_direct_insert=True as its tell. It is the recognizable symptom of picking the tier from the name rather than the axis.

Happy to open a PR for these three.

[gtouloumes]

Weighing in slightly late - but I like where this has landed from the initial proposal. Getting all four tiers on one consistent axis is a net benefit for new DataJoint users. Already feels like a change that would have helped me internalize DataJoint concepts more easily when first learning them. The "entry script" also helps with the ingestion overlap (even if I'd consider tweaking the term itself - see below).

A few thoughts for your consideration:

Weighing Derive vs Compute

We already describe this tier as the one that "derives from other tables" in the axis explanation itself — Derive just promotes that existing word to the class name, rather than introducing "compute," which carries HPC/resource baggage ("out of compute," compute nodes) that carries more overlap for our audience than for a general one. It does lose the semantic explanation of "computation" in the general sense, which is a fair downside.

Flagging "Entry script" collisions

The docs already use "entry point" for the tables themselves, and you note Manual/Imported are "entry-point tables" in the fan-out example. So, "entry script" for the writer isn't very far same root word in the same docs. It also mainly covers the automated half of your own definition, "a person, an instrument, an entry script", but a person isn't running a script, so hand-entry stays unnamed. It's not elegant, but "Loader script" could address both concerns (albeit with its own pre-existing connotations as a UI button, utility, etc).

Accepting Verb-as-class-name adjustment period

The grammar argument for Ingest/Compute is something I can get behind. Participles do seem to put the name on the wrong axis, which is the same flaw as Manual/Imported today. But Python class names are read as nouns almost universally, including Lookup and Entry in this same set. class TuningCurve(dj.Compute): will take real getting-used-to independent of whether the underlying logic is correct. Consider it an adoption cost, not a correctness objection, that may be worth reinforcing throughout the docs in the upcoming updates.

[dimitri-yatsenko]

Renamed to foreground the rename, now tracked across three repos:

• datajoint-python#1546 — introduce dj.Entry as the primary class, dj.Manual a permanent alias (implementation).
• datajoint-company/dj-brand#34 — adopt "Entry" as the canonical terminology (editorial rules).

The near-term docs asks (explain the tier axis; items 1–3 above) proceed here regardless of the rename decision.

[dimitri-yatsenko]

Thanks, @gtouloumes — glad the axis framing lands, and good pushback. On Derive vs Compute I'd keep Compute, for a database-specific reason.

In relational databases derived is already a loaded word: a derived table (derived relation) is the result of a query — a subquery or view, computed on read and not stored. That is the opposite of what this tier is: a computed table holds materialized rows produced by make() and persisted with lineage. Naming it Derive would collide head-on with the entrenched meaning for exactly our most database-literate readers, who would read "derived" as "a view / query result" — the wrong mental model for a table whose rows are computed once and kept.

Compute also aligns with how we already describe DataJoint — a computational database, where computation is a first-class, materialized part of the schema rather than a read-time projection. Keeping the tier name in that vocabulary reinforces the positioning instead of fighting it.

The HPC connotation you flag ("out of compute," compute nodes) is real, but it's ambient domain baggage that context disambiguates. The derived-relation collision is a precise, in-domain false friend that actively mismodels the tier — so between the two, Derive is the more misleading choice.

(The class-naming decision itself lives in datajoint/datajoint-python#1546 — I'll carry this rationale there. Your "loader/entry script" naming and the verb-as-class-name adoption-cost points are noted for that thread too; this issue stays about explaining the axis in the docs.)

[dimitri-yatsenko]

A second terminology shift the axis argument implies: retire "tier"

This issue's core observation is that the four names should sit on one axis — what puts rows in this table. Applying that test to the word tier itself suggests it should go too.

"Tier" names a rank, and these are not ranked. A tier is a level in an ordered arrangement — tier 1 support, tiered pricing, storage tiers where the ordering is the whole point. Lookup, Entry, Ingest and Compute are not ordered. Nothing is above or below anything. They are four answers to one question, which is a classification, not a hierarchy. The word imports an ordering that does not exist and then leaves the reader to discover it is spurious.

It is also a term the reader has to acquire before receiving the point. "The tier is the provenance classification" spends a definition before it says anything. "DataJoint marks every table by its origin" says it outright, and needs no glossary entry.

Suggested wording: origin, or table origin where a noun phrase is needed.

• "Every table declares its origin: Lookup, Entry, Ingest, or Compute."
• Reference tables become "the four origins" rather than "the five tiers."
• dj.Diagram's legend row Tables needs no change; the swatch keys the four names, which speak for themselves once they sit on one axis.

What this costs. Two things, and both are real:

1. Tier is a decade old and appears throughout the docs, the library (_TIER_STRUCTURE, _DIAGRAM_THEMES), the diagram guide, and third-party material. Prose can move without touching identifiers, the same way dj.Manual survives as an alias — but the split is a cost, and someone reading _TIER_STRUCTURE while the docs say "origin" has to bridge it.
2. Part does not fit the axis, and does not fit origin either. A part inherits its master's origin and has none of its own. The current docs paper over this by counting it as a fifth tier; under origin it needs stating plainly — a part is not a fifth origin, it is a structural role. That is arguably clearer than what we have, but it is extra prose rather than a free rename.

Sequencing: independent of the class rename, and cheaper. It is prose only, needs no release, and does not wait on a datajoint-python decision — which makes it a reasonable thing to do alongside docs items 1–3 rather than after them. If the axis is worth stating once where the names are introduced, that is the same paragraph where tier would otherwise have to be defined.

Surfaced from writing an essay that needed the distinction for a general reader: the sentence introducing "tier" was the only place the prose asked for vocabulary it did not otherwise need, and it read as jargon precisely because the ordering it implies is not there.