evaluate_coho_bkfram_check_file
During the backwards FRAM process (“bkfram”) for Coho, FRAM solves
for stock escapements numerically by iterating, (hopefully) ending with
a database where the “Escapement” column of the “Escapement” table
matches the corresponding rows of the “TargetEscAge3” column of the
“BackwardsFRAM” table. If this process fails to converge, it’s useful to
know which stocks were the problem, and what the attempted convergene
looks like. FRAM produces a “BackFramCheck.Txt” file storing the
convergence information, and framrsquared has several
functions to help understand that file. This vignette demonstrates
several of them. For more details see the help files for individual
functions.
Quick technical note: the ggrepel and readr
packages are optional for the framrsquared package, but are
necessary for the functions in this vignette. If you don’t already have
them installed, using the functions here will prompt you to install
them.
Basics of “BackFramCheck.Txt”
For testing and demonstration purposes, framrsquared
contains a demo “BackFramCheck.Txt” file. The filepath to it can be
found with system.file().
## [1] "/tmp/RtmpxMl4XI/Rinst102269c8387d/framrsquared.dev/BackFramCheck.Txt"The text file itself is a fixed-width-ish file with some metadata in
the first few lines. “fixed-width-ish” because
NaN entries are placed differently from all other entries
of a column.
## Backwards FRAM Iteration Calculations C:\Repos\BKFRAM TEST DB\BKFRAM TEST DB\bkCoho2023_A_2_b_TEST3 7/30/2026 12:00:00 AM
## bkCoho2023_A_2_b_TEST3 -Date- 7/30/2026 12:00:00 AM
##
## It# Stk Escape EscTarg EscTarg/Esc Old Scalar New Scalar StartCohort StockName
## 1 1 10590691 1474 0.0001 1000.0000 0.3868 11513849 - U-nkskrw
## 1 2 0 0 - 0.0000 0.0000 * - M-nkskrw
## 1 3 58691587 442 0.0000 1000.0000 0.2555 63807566 - U-kendlh
## 1 4 58671490 3331 0.0001 1000.0000 0.6470 63804968 - M-kendlh
## 1 5 85834356 245 0.0000 1000.0000 0.2505 93316270 - U-skokmh
## 1 6 85804981 14448 0.0002 1000.0000 0.7581 93312471 - M-skokmh
## 1 7 29 0 0.0000 0.0014 0.0016 * - U-lumpdh
## 1 8 8541 0 0.0000 0.4214 0.4862 * - M-lumpdh
## 1 9 0 0 - 0.0000 0.0000 * - U-bhambh
## 1 10 0 0 - 0.0000 0.0000 * - M-bhambh
## 1 11 23204922 1157 0.0000 1000.0000 0.2674 25226862 - U-samshw
## 1 12 0 0 - 0.0000 0.0000 * - M-samshw
## 1 13 860 0 0.0000 1.1171 1.2694 * - U-ar77aw
## 1 14 0 0 - 0.0000 0.0000 * - M-ar77aw
## 1 15 0 0 - 0.0000 0.0000 * - U-whatch
## 1 16 0 0 - 0.0000 0.0000 * - M-whatchThe most useful tools for examining convergence are our plotting
functions (below). However, if we are writing code to work with the
convergence data itself, parse_bkfram_check() or
process_bkfram_check() will be useful.
parse_bkfram_check() just reads in the dataframe looking
part of the text file, parsing it correctly, giving useful column names,
and converting *s, -s, and NaNs
to NA.
## # A tibble: 20 × 9
## iteration stock_id escapement escapement_target esc_target_ratio old_scalar
## <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
## 1 1 1 10590691 1474 0.0001 1000
## 2 1 2 0 0 NA 0
## 3 1 3 58691587 442 0 1000
## 4 1 4 58671490 3331 0.0001 1000
## 5 1 5 85834356 245 0 1000
## 6 1 6 85804981 14448 0.0002 1000
## 7 1 7 29 0 0 0.0014
## 8 1 8 8541 0 0 0.421
## 9 1 9 0 0 NA 0
## 10 1 10 0 0 NA 0
## 11 1 11 23204922 1157 0 1000
## 12 1 12 0 0 NA 0
## 13 1 13 860 0 0 1.12
## 14 1 14 0 0 NA 0
## 15 1 15 0 0 NA 0
## 16 1 16 0 0 NA 0
## 17 1 17 82748401 50047 0.0006 1000
## 18 1 18 0 0 NA 0
## 19 1 19 13255977 1677 0.0001 1000
## 20 1 20 13253666 24306 0.0018 1000
## # ℹ 3 more variables: new_scalar <dbl>, starting_cohort <dbl>, stock_name <chr>Typically we want to combine marked and unmarked components of
stocks. For any stock with a TargetFlag of 2 in the “BackwardsFRAM”
table, backwards FRAM solves for the joint escapement of marked and
unmarked stocks. If we don’t combine marked and unmarked in those cases,
the measures of convergence can appear to have problems that aren’t
actually there. The plotting functions below all handle this
appropriately by default, but we may want the aggregated numbers
themselves for other scripting. process_bkfram_check()
provides these:
## # A tibble: 10 × 11
## iteration stock_aggregate_name stock_aggregate_id escapement
## <dbl> <chr> <dbl> <dbl>
## 1 1 nkskrw 1 10590691
## 2 1 kendlh 2 117363077
## 3 1 skokmh 3 171639337
## 4 1 lumpdh 4 8570
## 5 1 bhambh 5 0
## 6 1 samshw 6 23204922
## 7 1 ar77aw 7 860
## 8 1 whatch 8 0
## 9 1 skagtw 9 82748401
## 10 1 skagth 10 26509643
## # ℹ 7 more variables: escapement_target <dbl>, starting_cohort <dbl>,
## # stock_id <list>, stock_name <list>, missing_start_cohort <lgl>,
## # esc_target_ratio <dbl>, stock_label <chr>## Rows: 12,177
## Columns: 11
## $ iteration <dbl> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1…
## $ stock_aggregate_name <chr> "nkskrw", "kendlh", "skokmh", "lumpdh", "bhambh",…
## $ stock_aggregate_id <dbl> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15…
## $ escapement <dbl> 10590691, 117363077, 171639337, 8570, 0, 23204922…
## $ escapement_target <dbl> 1474, 3773, 14693, 0, 0, 1157, 0, 0, 50047, 25983…
## $ starting_cohort <dbl> NA, 127612534, 186628741, NA, NA, NA, NA, NA, NA,…
## $ stock_id <list> <1, 2>, <3, 4>, <5, 6>, <7, 8>, <9, 10>, <11, 12…
## $ stock_name <list> <"U-nkskrw", "M-nkskrw">, <"U-kendlh", "M-kendlh…
## $ missing_start_cohort <lgl> TRUE, FALSE, FALSE, TRUE, TRUE, TRUE, TRUE, TRUE,…
## $ esc_target_ratio <dbl> 1.391788e-04, 3.214810e-05, 8.560392e-05, 0.00000…
## $ stock_label <chr> "nkskrw (1, 2)", "kendlh (3, 4)", "skokmh (5, 6)"…The stock_id and stock_name columns are
actually list columns, with each row containing both the stock ids and
both the stock names of the two stocks that have been combined. See
?process_bkfram_check() for details.
process_bkfram_check() also does a few minor processing
steps (e.g., creating labels for easier plotting), and can optionally do
those additional steps without combining marked and unmarked stocks.
Diagnostic Plots
BIG NOTICE: bkfram usually converges! So with the example files, the default plotting options of the first two functions will show very boring plots! These functions optionally allow us to provide an iteration number to plot. If we don’t, the final iteration is plotted (with hopefully everything matching perfectly). In the examples below, I will focus on showing a plot of the early iteration, because that’s more useful for understanding the functions. When using these functions in the wild, you’ll probably want to start with the default behavior, since we don’t expect convergence in early iterations.
Which stocks haven’t converged? Barplot edition.
plot_bkfram_convergence_bar() highlights stocks for
which the model escapement value and the target escapement value aren’t
matching. By default, it plots the ratio of target/model escpement:
We can see that in this case the Tulalip stocks (33, 34) are over the target by a bit, and the Kendal hatchery stocks (3, 4) are about 2/3 of the target.
We can adjust the cutoff threshold for including a stock in the plot
by changing the thresh argument. The default value is 0.01.
Let’s filter to just those with ratios < 0.95 or > 1.05.
We can also plot and filter based on the difference in fish between target and model escapement instead of the ratio of target and model escapement. When this is the case, we likely want to change the threshold, since a threshold of 0.01 fish is a bit extreme.
If all stocks have converged within our threshold by the iteration in question, we get the following:
## ✔ No stocks outside of 0.01 of 1:1 ratio by iteration 60!
When iteration is not provided, the final iteration is
used. If the backwards run converged, this will always give the “All
escapements converged” version of the plot:
## ✔ No stocks outside of 0.01 of 1:1 ratio by iteration 99!
Which stocks haven’t converged? Scatterplot edition.
plot_bkfram_convergence_scatter() provides an
alternative view of non-convergence. It again filters to stocks that
haven’t converged (as definied by the target/model escapement being at
least thresh away from 1), but these stocks by their model
and target escapement with labels. Because the plot labeling function
(from the ggrepel package) can cause plotting to bog down
if there are many points, if there are more than n_max
stocks that haven’t converged (default: 10), only the n_max
stocks with the most extreme ratios are plotted. Here deviation from the
1:1 line (the dashed line) represents how far the stocks are from
convergence.
Just for fun, we can see how the convergence process progresses over
time, using the package patchwork to combine figures. I’ll
choose iterations so there’s visible progress but we don’t immediately
jump to perfect convergence.
library(patchwork)
library(ggplot2)
## save plots to objects
## Overwriting titles for simpler reading.
gp4 <- plot_bkfram_convergence_scatter(filepath = filepath, iteration = 4, label_size = 2.6)+
labs(subtitle = "Iteration 4", title = "", y = " model escapement")## ! 12 stocks non-converged by iteration 4! Plotting most extreme 10.gp6 <- plot_bkfram_convergence_scatter(filepath = filepath, iteration = 6, label_size = 2.6)+
labs(subtitle = "Iteration 6", title = "", y = " model escapement")
gp8 <- plot_bkfram_convergence_scatter(filepath = filepath, iteration = 8, label_size = 2.6) +
labs(subtitle = "Iteration 8", title = "", y = " model escapement")
gp10 <- plot_bkfram_convergence_scatter(filepath = filepath, iteration = 10, label_size = 2.6)+
labs(subtitle = "Iteration 10", title = "", y = " model escapement")
## patchwork nomenclature to lay out plots, combine identical axis titles
gp4 + gp6 + gp8 + gp10 + plot_layout(ncol = 2, axis_titles = "collect")
Looking across these from low to high iterations, we can see that (a) decreasing numbers of stocks have failed to converge, and (b) the scale of our plots zooms in, so that deviations from the 1:1 line represent much smaller deviations from convergence.
But with these plots it’s hard to track how convergence works for an
individaul stock. For example, what’s happening with the
kendlh stocks (stock id 3, 4)? Well…
Tracing individual stocks
plot_bkfram_convergence_trace() plots the convergence of
one or more stocks, and overlays the targets. Unlike the functions
above, we need to provide one or more stock IDs.
So long as optional argument aggregate_stocks is
TRUE (the default value), we only need to include a single
id from the the mark-unmarked pair. That is, we get the same plot if we
just provide one of the ids:
We can provide more stock ids to show the convergence of several stocks at once.
Suggested application:
Did bkfram converge? If so, you probably don’t need any of these.
However, you might enjoy playing around with any of these functions to
better understand how FRAM approaches convergence. If you’re doing so,
you’ll want to use the optional iteration argument for
plot_bkfram_convergence_bar() and
plot_bkfram_convergence_scatter() to plot the process in
early iterations before things have converged.
If bkfram didn’t converge, try
plot_bkfram_convergence_bar() or
plot_bkfram_convergence_scatter(), providing just the
filepath. This will show you which stocks failed to converge. You can
tweak the thresh argument to make the plotting more or less
conservative. If you’re tweaking thresh, you may want to
use plot_bkfram_convergence_bar() with
plot_ratio = FALSE so that your thresh cutoff
is in numbers of actual fish.
If you’ve identified one or more stocks that haven’t converged, you
may want to look at how they’ve behaved across iterations, which you can
do with plot_bkfram_convergence_trace(). The stock names
from the bar and scatter plots will include the stock ids; you can use
those for the stock_id argument in *_trace().
If the trace shows the model escapements slowly approaching their
targets, bkfram may just need more iterations. You can start a second
run that starts with the solution of the first run to give more
iterations.