Repository navigation
Cache expensive reflection calls #2401
Description
Is your feature request related to a problem? Please describe.
I'd like to use pgx.RowToStructByName instead of scany, but currently, it regenerates the struct scan definition on every call. Since the number of structs in a program is limited, we should cache these definitions.
Describe the solution you'd like
Cache the fields for each type in the following code to avoid generating them every time.
Lines 685 to 694 in a2fca03
Describe alternatives you've considered
Keep using scany
Additional context
Refs:
The patch to change from scany to pgx with a sample bench test
diff --git a/pkg/search/postgres/select.go b/pkg/search/postgres/select.go
index 6b25478c04..4cd05f66ec 100644
--- a/pkg/search/postgres/select.go
+++ b/pkg/search/postgres/select.go
@@ -7,8 +7,7 @@ import (
"strings"
"time"
- "github.com/georgysavva/scany/v2/dbscan"
- "github.com/georgysavva/scany/v2/pgxscan"
+ "github.com/jackc/pgx/v5"
"github.com/pkg/errors"
v1 "github.com/stackrox/rox/generated/api/v1"
"github.com/stackrox/rox/pkg/env"
@@ -19,27 +18,8 @@ import (
"github.com/stackrox/rox/pkg/search/paginated"
"github.com/stackrox/rox/pkg/search/postgres/aggregatefunc"
pgsearch "github.com/stackrox/rox/pkg/search/postgres/query"
- "github.com/stackrox/rox/pkg/utils"
)
-var scanAPI = newScanAPI(newDBScanAPI(dbscan.WithAllowUnknownColumns(true)))
-
-func newScanAPI(dbscanAPI *dbscan.API) *pgxscan.API {
- api, err := pgxscan.NewAPI(dbscanAPI)
- if err != nil {
- utils.Must(err)
- }
- return api
-}
-
-func newDBScanAPI(opts ...dbscan.APIOption) *dbscan.API {
- api, err := pgxscan.NewDBScanAPI(opts...)
- if err != nil {
- utils.Must(err)
- }
- return api
-}
-
// RunSelectRequestForSchema executes a select request against the database for given schema. The input query must
// explicitly specify select fields.
// Deprecated: Use RunSelectRequestForSchemaFn
@@ -160,10 +140,9 @@ func retryableRunSelectRequestForSchemaFn[T any](ctx context.Context, db postgre
}
defer rows.Close()
- scanner := scanAPI.NewRowScanner(rows)
for rows.Next() {
- var row T
- if err := scanner.Scan(&row); err != nil {
+ row, err := pgx.RowToStructByName[T](rows)
+ if err != nil {
return err
}
if err := fn(&row); err != nil { │ scany.txt │ pgx.txt │
│ max_heap_bytes │ max_heap_bytes vs base │
EvaluateBaselinesAndPersistResult/100_processes_2_containers-8 11.58M ± 8% 12.64M ± 55% +9.16% (p=0.015 n=6)
EvaluateBaselinesAndPersistResult/500_processes_3_containers-8 14.37M ± 24% 14.45M ± 7% ~ (p=0.937 n=6)
EvaluateBaselinesAndPersistResult/1000_processes_5_containers-8 15.94M ± 24% 16.31M ± 24% ~ (p=0.485 n=6)
EvaluateBaselinesAndPersistResult/1000_processes_5_containers_large_args-8 21.57M ± 23% 21.54M ± 19% ~ (p=0.937 n=6)
EvaluateBaselinesAndPersistResult/2000_processes_10_containers-8 19.93M ± 17% 21.26M ± 3% ~ (p=0.394 n=6)
EvaluateBaselinesAndPersistResult/10000_processes_20_containers-8 31.26M ± 25% 42.44M ± 9% +35.75% (p=0.004 n=6)
EvaluateBaselinesAndPersistResult/25000_processes_50_containers-8 41.02M ± 35% 53.73M ± 12% ~ (p=0.132 n=6)
EvaluateBaselinesAndPersistResult/50000_processes_100_containers-8 38.95M ± 46% 67.87M ± 34% +74.25% (p=0.004 n=6)
EvaluateBaselinesAndPersistResult/50000_processes_100_containers_large_args-8 80.47M ± 17% 92.02M ± 11% ~ (p=0.093 n=6)
geomean 25.52M 29.96M +17.41%
│ master.txt │ new.txt │
│ max_heap_objects │ max_heap_objects vs base │
EvaluateBaselinesAndPersistResult/100_processes_2_containers-8 11.58M ± 8% 12.64M ± 55% +9.16% (p=0.015 n=6)
EvaluateBaselinesAndPersistResult/500_processes_3_containers-8 14.37M ± 24% 14.45M ± 7% ~ (p=0.937 n=6)
EvaluateBaselinesAndPersistResult/1000_processes_5_containers-8 15.94M ± 24% 16.31M ± 24% ~ (p=0.485 n=6)
EvaluateBaselinesAndPersistResult/1000_processes_5_containers_large_args-8 21.57M ± 23% 21.54M ± 19% ~ (p=0.937 n=6)
EvaluateBaselinesAndPersistResult/2000_processes_10_containers-8 19.93M ± 17% 21.26M ± 3% ~ (p=0.394 n=6)
EvaluateBaselinesAndPersistResult/10000_processes_20_containers-8 31.26M ± 25% 42.44M ± 9% +35.75% (p=0.004 n=6)
EvaluateBaselinesAndPersistResult/25000_processes_50_containers-8 41.02M ± 35% 53.73M ± 12% ~ (p=0.132 n=6)
EvaluateBaselinesAndPersistResult/50000_processes_100_containers-8 38.95M ± 46% 67.87M ± 34% +74.25% (p=0.004 n=6)
EvaluateBaselinesAndPersistResult/50000_processes_100_containers_large_args-8 80.47M ± 17% 92.02M ± 11% ~ (p=0.093 n=6)
geomean 25.52M 29.96M +17.41%
│ master.txt │ new.txt │
│ B/op │ B/op vs base │
EvaluateBaselinesAndPersistResult/100_processes_2_containers-8 660.8Ki ± 0% 938.6Ki ± 0% +42.04% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/500_processes_3_containers-8 2.031Mi ± 0% 3.400Mi ± 0% +67.38% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/1000_processes_5_containers-8 3.616Mi ± 0% 6.354Mi ± 0% +75.73% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/1000_processes_5_containers_large_args-8 8.794Mi ± 0% 11.536Mi ± 0% +31.19% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/2000_processes_10_containers-8 6.680Mi ± 0% 12.168Mi ± 0% +82.14% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/10000_processes_20_containers-8 31.70Mi ± 0% 59.16Mi ± 0% +86.61% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/25000_processes_50_containers-8 77.96Mi ± 0% 146.62Mi ± 0% +88.07% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/50000_processes_100_containers-8 155.1Mi ± 0% 292.4Mi ± 0% +88.57% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/50000_processes_100_containers_large_args-8 442.4Mi ± 0% 579.7Mi ± 0% +31.02% (p=0.002 n=6)
geomean 15.35Mi 25.19Mi +64.16%
│ master.txt │ new.txt │
│ allocs/op │ allocs/op vs base │
EvaluateBaselinesAndPersistResult/100_processes_2_containers-8 15.45k ± 0% 16.39k ± 0% +6.09% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/500_processes_3_containers-8 57.83k ± 0% 62.76k ± 0% +8.53% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/1000_processes_5_containers-8 108.5k ± 0% 118.4k ± 0% +9.12% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/1000_processes_5_containers_large_args-8 108.5k ± 0% 118.5k ± 0% +9.16% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/2000_processes_10_containers-8 208.5k ± 0% 228.4k ± 0% +9.56% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/10000_processes_20_containers-8 1.015M ± 0% 1.115M ± 0% +9.85% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/25000_processes_50_containers-8 2.515M ± 0% 2.765M ± 0% +9.94% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/50000_processes_100_containers-8 5.015M ± 0% 5.515M ± 0% +9.97% (p=0.002 n=6)
EvaluateBaselinesAndPersistResult/50000_processes_100_containers_large_args-8 5.016M ± 0% 5.516M ± 0% +9.96% (p=0.002 n=6)
geomean 373.3k 407.4k +9.12%
Activity
[-]Cache expensive reflection call when scan to struct by name[/-][+]Cache expensive reflection calls[/+]jackc commentedon Oct 11, 2025
There already is some caching being used. See the usage of
namedStructFieldMapin rows.go. I'm definitely open to additional improvements though.janisz commentedon Oct 29, 2025
So far I found https://github.com/zolstein/pgx-collect