Dataset: Average Across Datasets
Key Metrics:
packed 5.5k at 0.334packed 20k at 0.4459packed_fixed 1k at 0.4091file at 0% (0 per 100k lines)packed 5.5k at 109.0088s| Test Configuration | F1 | Precision | Recall | Skipped (%) | Skipped Lines / 100k | Processing Time (s) |
|---|---|---|---|---|---|---|
| project | 0.2779 | 0.4862 | 0.2035 | 27.0793% | 0.0196 | 164.906 |
| file | 0.2845 | 0.2689 | 0.3177 | 0% | 0 | 175.6137 |
| original 0.5k | 0.2658 | 0.2306 | 0.3336 | 0% | 0 | 179.2182 |
| original 0.7k | 0.3017 | 0.2695 | 0.3605 | 0% | 0 | 171.301 |
| original 1k | 0.2873 | 0.2541 | 0.3479 | 0% | 0 | 171.7211 |
| original 1.3k | 0.2844 | 0.2514 | 0.3467 | 0% | 0 | 168.927 |
| original 1.6k | 0.2814 | 0.2444 | 0.3472 | 0% | 0 | 171.9007 |
| syntactic 1k | 0.2485 | 0.1906 | 0.3662 | 0% | 0 | 288.5075 |
| syntactic 3k | 0.2541 | 0.1936 | 0.3802 | 0% | 0 | 243.2186 |
| syntactic 5k | 0.2506 | 0.1955 | 0.3624 | 0% | 0 | 244.0636 |
| syntactic 6k | 0.2616 | 0.2062 | 0.3691 | 0% | 0 | 238.5688 |
| syntactic 7k | 0.2638 | 0.208 | 0.3732 | 0% | 0 | 240.1502 |
| syntactic 10k | 0.2506 | 0.1937 | 0.3665 | 0% | 0 | 237.7785 |
| syntactic 13k | 0.2622 | 0.2089 | 0.3659 | 0.3915% | 0.0003 | 241.7863 |
| syntactic 15k | 0.2498 | 0.1935 | 0.3631 | 0% | 0 | 237.3187 |
| syntactic 17k | 0.2599 | 0.2063 | 0.3661 | 0% | 0 | 237.2653 |
| fixed_token 1k | 0.2305 | 0.1679 | 0.3695 | 0.3961% | 0.0003 | 289.3473 |
| fixed_token 3k | 0.2528 | 0.2084 | 0.3245 | 0% | 0 | 195.9924 |
| fixed_token 5k | 0.2786 | 0.2521 | 0.325 | 0.9211% | 0.0007 | 184.1849 |
| fixed_token 5.5k | 0.2755 | 0.2498 | 0.3224 | 0% | 0 | 172.7631 |
| fixed_token 10k | 0.294 | 0.2632 | 0.3417 | 0% | 0 | 175.555 |
| fixed_token 13k | 0.2901 | 0.2719 | 0.3246 | 0.9211% | 0.0007 | 202.8863 |
| fixed_token 20k | 0.2818 | 0.2647 | 0.3139 | 0% | 0 | 177.2859 |
| fixed_token 100k | 0.2766 | 0.2571 | 0.3171 | 0.9211% | 0.0007 | 205.0634 |
| packed 1k | 0.2395 | 0.1911 | 0.3301 | 0.3915% | 0.0003 | 248.1151 |
| packed 3k | 0.2811 | 0.2683 | 0.2962 | 0.5342% | 0.0004 | 154.7025 |
| packed 4k | 0.2994 | 0.2936 | 0.3075 | 0% | 0 | 218.478 |
| packed 5.5k | 0.334 | 0.3468 | 0.3343 | 0% | 0 | 109.0088 |
| packed 6k | 0.2872 | 0.2862 | 0.297 | 0% | 0 | 255.8858 |
| packed 6.5k | 0.3122 | 0.3418 | 0.2878 | 0% | 0 | 111.7946 |
| packed 7k | 0.3062 | 0.3248 | 0.2916 | 1.7546% | 0.0013 | 216.5816 |
| packed 10k | 0.294 | 0.344 | 0.259 | 6.2494% | 0.0045 | 173.4338 |
| packed 11k | 0.2844 | 0.3827 | 0.2288 | 0% | 0 | 119.9188 |
| packed 12k | 0.3076 | 0.3903 | 0.2696 | 0% | 0 | 109.9375 |
| packed 13k | 0.2876 | 0.5236 | 0.1995 | 7.479% | 0.0054 | 328.4267 |
| packed 15k | 0.2693 | 0.3224 | 0.2326 | 8.8837% | 0.0064 | 412.2806 |
| packed 17k | 0.2836 | 0.4101 | 0.2175 | 10.3251% | 0.0075 | 461.406 |
| packed 20k | 0.2883 | 0.4459 | 0.2181 | 0% | 0 | 118.3206 |
| packed 23k | 0.3002 | 0.4315 | 0.2329 | 3.2974% | 0.0024 | 213.3169 |
| packed 25k | 0.3184 | 0.405 | 0.2638 | 7.4468% | 0.0054 | 317.9901 |
| packed 30k | 0.2618 | 0.44 | 0.1961 | 18.6792% | 0.0135 | 261.0893 |
| packed 40k | 0.2741 | 0.4553 | 0.2028 | 24.3622% | 0.0176 | 303.8802 |
| packed_fixed 1k | 0.2407 | 0.1719 | 0.4091 | 0.6171% | 0.0004 | 309.8403 |
| packed_fixed 3k | 0.2822 | 0.242 | 0.3455 | 0.875% | 0.0006 | 178.7686 |
| packed_fixed 4k | 0.2839 | 0.2802 | 0.2969 | 1.8421% | 0.0013 | 161.9185 |
| packed_fixed 5.5k | 0.2874 | 0.3016 | 0.2796 | 1.0638% | 0.0008 | 126.7866 |
| packed_fixed 5k | 0.3201 | 0.319 | 0.3273 | 0.9211% | 0.0007 | 149.3238 |
| packed_fixed 6k | 0.2756 | 0.2648 | 0.2914 | 0.9211% | 0.0007 | 151.0538 |
| packed_fixed 7k | 0.2906 | 0.3197 | 0.2721 | 1.5428% | 0.0011 | 168.1952 |
| packed_fixed 8k | 0.2943 | 0.3535 | 0.2675 | 3.0856% | 0.0022 | 132.6804 |
| packed_fixed 10k | 0.274 | 0.3342 | 0.2374 | 1.8421% | 0.0013 | 145.7211 |
| packed_fixed 11k | 0.3157 | 0.3469 | 0.2955 | 0% | 0 | 167.7527 |
| packed_fixed 12k | 0.3039 | 0.3858 | 0.2578 | 4.3014% | 0.0031 | 335.7311 |
| packed_fixed 13k | 0.2927 | 0.3902 | 0.2386 | 0% | 0 | 122.3278 |
| packed_fixed 15k | 0.3121 | 0.405 | 0.2575 | 8.7777% | 0.0064 | 312.4827 |
| packed_fixed 17k | 0.2901 | 0.3609 | 0.2489 | 11.5778% | 0.0084 | 274.44 |
| packed_fixed 20k | 0.2635 | 0.4047 | 0.1991 | 0% | 0 | 136.2043 |
| packed_fixed 23k | 0.2474 | 0.4699 | 0.1693 | 13.4844% | 0.0098 | 171.7707 |
| packed_fixed 25k | 0.2882 | 0.4862 | 0.2106 | 9.1462% | 0.0066 | 224.8978 |
| packed_fixed 30k | 0.2978 | 0.4678 | 0.2235 | 7.8981% | 0.0057 | 214.1122 |
| packed_fixed 40k | 0.2331 | 0.5518 | 0.1543 | 4.5731% | 0.0033 | 177.7168 |
Dataset: repos/juice-shop
Key Metrics:
fixed_token 20k at 0.2687packed_fixed 20k at 0.3696fixed_token 1k at 0.2766file at 0% (0 per 100k lines)packed_fixed 20k at 99.495s| Test Configuration | F1 | Precision | Recall | Skipped (%) | Skipped Lines / 100k | Processing Time (s) |
|---|---|---|---|---|---|---|
| project | 0.25 | 0.375 | 0.1875 | 81.2379% | 0.0588 | 192.1778 |
| file | 0.2564 | 0.2809 | 0.2358 | 0% | 0 | 203.0092 |
| original 0.5k | 0.1782 | 0.1978 | 0.1622 | 0% | 0 | 213.8311 |
| original 0.7k | 0.2189 | 0.2391 | 0.2018 | 0% | 0 | 201.0304 |
| original 1k | 0.2079 | 0.2258 | 0.1927 | 0% | 0 | 210.574 |
| original 1.3k | 0.22 | 0.2418 | 0.2018 | 0% | 0 | 196.2363 |
| original 1.6k | 0.2075 | 0.2157 | 0.2 | 0% | 0 | 217.3364 |
| syntactic 1k | 0.2055 | 0.1677 | 0.2653 | 0% | 0 | 301.43 |
| syntactic 3k | 0.189 | 0.1538 | 0.2449 | 0% | 0 | 313.4625 |
| syntactic 5k | 0.2023 | 0.1635 | 0.2653 | 0% | 0 | 312.2644 |
| syntactic 6k | 0.2122 | 0.1745 | 0.2708 | 0% | 0 | 301.3921 |
| syntactic 7k | 0.2041 | 0.1701 | 0.2551 | 0% | 0 | 315.4044 |
| syntactic 10k | 0.1882 | 0.1538 | 0.2424 | 0% | 0 | 302.2646 |
| syntactic 13k | 0.2083 | 0.1761 | 0.2551 | 1.1744% | 0.0008 | 322.1633 |
| syntactic 15k | 0.1938 | 0.1553 | 0.2577 | 0% | 0 | 306.545 |
| syntactic 17k | 0.1912 | 0.1579 | 0.2424 | 0% | 0 | 306.3632 |
| fixed_token 1k | 0.1891 | 0.1436 | 0.2766 | 1.1882% | 0.0009 | 403.0953 |
| fixed_token 3k | 0.2353 | 0.2097 | 0.268 | 0% | 0 | 242.4231 |
| fixed_token 5k | 0.2412 | 0.2526 | 0.2308 | 2.7632% | 0.002 | 237.1015 |
| fixed_token 5.5k | 0.2538 | 0.2778 | 0.2336 | 0% | 0 | 219.3301 |
| fixed_token 10k | 0.2551 | 0.2632 | 0.2475 | 0% | 0 | 203.0964 |
| fixed_token 13k | 0.264 | 0.2826 | 0.2476 | 2.7632% | 0.002 | 291.6924 |
| fixed_token 20k | 0.2687 | 0.2872 | 0.2523 | 0% | 0 | 218.6027 |
| fixed_token 100k | 0.2462 | 0.2667 | 0.2286 | 2.7632% | 0.002 | 295.3265 |
| packed 1k | 0.1974 | 0.1797 | 0.219 | 1.1744% | 0.0008 | 356.9397 |
| packed 3k | 0.2264 | 0.2243 | 0.2286 | 1.6027% | 0.0012 | 209.1836 |
| packed 4k | 0.2626 | 0.2766 | 0.25 | 0% | 0 | 440.7919 |
| packed 5.5k | 0.24 | 0.2449 | 0.2353 | 0% | 0 | 116.6717 |
| packed 6k | 0.1969 | 0.2235 | 0.1759 | 0% | 0 | 131.641 |
| packed 6.5k | 0.2353 | 0.2651 | 0.2115 | 0% | 0 | 106.3249 |
| packed 7k | 0.2308 | 0.2727 | 0.2 | 5.2639% | 0.0038 | 413.7669 |
| packed 10k | 0.213 | 0.2687 | 0.1765 | 8.0962% | 0.0059 | 117.2829 |
| packed 11k | 0.2169 | 0.3158 | 0.1651 | 0% | 0 | 127.5833 |
| packed 12k | 0.2024 | 0.3036 | 0.1518 | 0% | 0 | 122.1906 |
| packed 13k | 0.2205 | 0.3684 | 0.1573 | 22.4371% | 0.0162 | 741.3708 |
| packed 15k | 0.2158 | 0.2885 | 0.1724 | 26.651% | 0.0193 | 968.3946 |
| packed 17k | 0.2188 | 0.3043 | 0.1707 | 30.9754% | 0.0224 | 1074.4166 |
| packed 20k | 0.2209 | 0.3065 | 0.1727 | 0% | 0 | 122.7695 |
| packed 23k | 0.1831 | 0.3333 | 0.1262 | 9.8922% | 0.0072 | 168.7102 |
| packed 25k | 0.259 | 0.3673 | 0.2 | 22.3404% | 0.0162 | 687.1106 |
| packed 30k | 0.1538 | 0.4286 | 0.0938 | 56.0376% | 0.0406 | 432.4359 |
| packed 40k | 0.2222 | 0.375 | 0.1579 | 73.0865% | 0.0529 | 514.9739 |
| packed_fixed 1k | 0.1838 | 0.142 | 0.2604 | 1.8513% | 0.0013 | 451.2084 |
| packed_fixed 3k | 0.1947 | 0.1897 | 0.2 | 2.625% | 0.0019 | 252.3721 |
| packed_fixed 4k | 0.2629 | 0.3108 | 0.2277 | 5.5264% | 0.004 | 257.8889 |
| packed_fixed 5.5k | 0.2093 | 0.2571 | 0.1765 | 3.6198% | 0.0026 | 174.988 |
| packed_fixed 5k | 0.2541 | 0.284 | 0.23 | 2.7632% | 0.002 | 223.3354 |
| packed_fixed 5.5k | 0.2328 | 0.2683 | 0.2056 | 2.7632% | 0.002 | 181.7377 |
| packed_fixed 6k | 0.2178 | 0.2316 | 0.2056 | 2.7632% | 0.002 | 235.8792 |
| packed_fixed 7k | 0.2081 | 0.2647 | 0.1714 | 4.6284% | 0.0034 | 289.6858 |
| packed_fixed 8k | 0.2452 | 0.3585 | 0.1863 | 9.2567% | 0.0067 | 208.6789 |
| packed_fixed 10k | 0.2346 | 0.3276 | 0.1827 | 5.5264% | 0.004 | 201.2664 |
| packed_fixed 11k | 0.2442 | 0.3231 | 0.1963 | 0% | 0 | 180.8581 |
| packed_fixed 12k | 0.2933 | 0.3667 | 0.2444 | 12.9041% | 0.0093 | 835.4004 |
| packed_fixed 13k | 0.2456 | 0.3387 | 0.1927 | 0% | 0 | 100.2038 |
| packed_fixed 15k | 0.2609 | 0.4 | 0.1935 | 26.3332% | 0.0191 | 718.1091 |
| packed_fixed 17k | 0.2276 | 0.359 | 0.1667 | 34.7334% | 0.0251 | 286.5238 |
| packed_fixed 20k | 0.2125 | 0.3696 | 0.1491 | 0% | 0 | 99.495 |
| packed_fixed 23k | 0.1835 | 0.2778 | 0.137 | 40.4532% | 0.0293 | 268.1211 |
| packed_fixed 25k | 0.2264 | 0.5 | 0.1463 | 27.4385% | 0.0199 | 240.736 |
| packed_fixed 30k | 0.224 | 0.4375 | 0.1505 | 23.6944% | 0.0171 | 221.6367 |
| packed_fixed 40k | 0.1732 | 0.4783 | 0.1058 | 13.7193% | 0.0099 | 215.6896 |
Dataset: repos/validation-benchmarks
Key Metrics:
packed 23k at 0.4675project at 0.6923syntactic 3k at 0.5385project at 0 per 100k linespacked 5.5k at 55.4401s| Test Configuration | F1 | Precision | Recall | Skipped (%) | Skipped Lines / 100k | Processing Time (s) |
|---|---|---|---|---|---|---|
| project | 0.2812 | 0.6923 | 0.1765 | 0% | 0 | 86.808 |
| file | 0.3051 | 0.24 | 0.4186 | 0% | 0 | 136.3079 |
| original 0.5k | 0.3333 | 0.2532 | 0.4878 | 0% | 0 | 127.3545 |
| original 0.7k | 0.3729 | 0.2895 | 0.5238 | 0% | 0 | 126.6157 |
| original 1k | 0.3621 | 0.28 | 0.5122 | 0% | 0 | 125.8866 |
| original 1.3k | 0.3415 | 0.2625 | 0.4884 | 0% | 0 | 131.1315 |
| original 1.6k | 0.3652 | 0.28 | 0.525 | 0% | 0 | 126.1201 |
| syntactic 1k | 0.2899 | 0.2041 | 0.5 | 0% | 0 | 179.2846 |
| syntactic 3k | 0.3066 | 0.2143 | 0.5385 | 0% | 0 | 186.1151 |
| syntactic 5k | 0.3043 | 0.2143 | 0.525 | 0% | 0 | 183.6036 |
| syntactic 6k | 0.3158 | 0.2258 | 0.525 | 0% | 0 | 194.4113 |
| syntactic 7k | 0.3188 | 0.2268 | 0.5366 | 0% | 0 | 187.6103 |
| syntactic 10k | 0.2857 | 0.2 | 0.5 | 0% | 0 | 179.2933 |
| syntactic 13k | 0.3134 | 0.2234 | 0.525 | 0% | 0 | 181.8644 |
| syntactic 15k | 0.3088 | 0.2188 | 0.525 | 0% | 0 | 183.8477 |
| syntactic 17k | 0.3182 | 0.2258 | 0.5385 | 0% | 0 | 188.4444 |
| fixed_token 1k | 0.274 | 0.1961 | 0.4545 | 0% | 0 | 174.9415 |
| fixed_token 3k | 0.281 | 0.2237 | 0.3778 | 0% | 0 | 136.2005 |
| fixed_token 5k | 0.3089 | 0.2405 | 0.4318 | 0% | 0 | 126.9208 |
| fixed_token 5.5k | 0.3193 | 0.2533 | 0.4318 | 0% | 0 | 128.2521 |
| fixed_token 10k | 0.3158 | 0.2571 | 0.4091 | 0% | 0 | 138.7798 |
| fixed_token 13k | 0.3077 | 0.2432 | 0.4186 | 0% | 0 | 133.3635 |
| fixed_token 20k | 0.2931 | 0.2329 | 0.3953 | 0% | 0 | 135.8236 |
| fixed_token 100k | 0.3 | 0.2308 | 0.4286 | 0% | 0 | 143.2694 |
| packed 1k | 0.2879 | 0.2159 | 0.4318 | 0% | 0 | 143.587 |
| packed 3k | 0.3579 | 0.3469 | 0.3696 | 0% | 0 | 88.9032 |
| packed 4k | 0.3564 | 0.3273 | 0.3913 | 0% | 0 | 83.0021 |
| packed 5.5k | 0.4675 | 0.5455 | 0.4091 | 0% | 0 | 55.4401 |
| packed 6k | 0.3488 | 0.3659 | 0.3333 | 0% | 0 | 88.1855 |
| packed 6.5k | 0.425 | 0.4722 | 0.3864 | 0% | 0 | 90.8224 |
| packed 7k | 0.4 | 0.4048 | 0.3953 | 0% | 0 | 73.3802 |
| packed 10k | 0.3855 | 0.4103 | 0.3636 | 0% | 0 | 88.2376 |
| packed 11k | 0.3951 | 0.4706 | 0.3404 | 0% | 0 | 74.2306 |
| packed 12k | 0.4444 | 0.4348 | 0.4545 | 0% | 0 | 68.1512 |
| packed 13k | 0.3944 | 0.6667 | 0.28 | 0% | 0 | 106.2519 |
| packed 15k | 0.2857 | 0.3333 | 0.25 | 0% | 0 | 108.3821 |
| packed 17k | 0.3514 | 0.4815 | 0.2766 | 0% | 0 | 98.6044 |
| packed 20k | 0.3784 | 0.5 | 0.3043 | 0% | 0 | 102.3161 |
| packed 23k | 0.4675 | 0.6207 | 0.375 | 0% | 0 | 167.1763 |
| packed 25k | 0.4054 | 0.4839 | 0.3488 | 0% | 0 | 140.1396 |
| packed 30k | 0.3333 | 0.4194 | 0.2766 | 0% | 0 | 193.3261 |
| packed 40k | 0.3333 | 0.4194 | 0.2766 | 0% | 0 | 191.3198 |
| packed_fixed 1k | 0.2838 | 0.1927 | 0.5385 | 0% | 0 | 178.107 |
| packed_fixed 3k | 0.3738 | 0.3175 | 0.4545 | 0% | 0 | 90.424 |
| packed_fixed 4k | 0.28 | 0.2545 | 0.3111 | 0% | 0 | 86.7035 |
| packed_fixed 5.5k | 0.3297 | 0.3488 | 0.3125 | 0% | 0 | 65.7517 |
| packed_fixed 5k | 0.4045 | 0.4091 | 0.4 | 0% | 0 | 69.0496 |
| packed_fixed 5.5k | 0.3721 | 0.4 | 0.3478 | 0% | 0 | 70.4831 |
| packed_fixed 6k | 0.3297 | 0.3191 | 0.3409 | 0% | 0 | 63.3355 |
| packed_fixed 7k | 0.4138 | 0.4615 | 0.375 | 0% | 0 | 60.3973 |
| packed_fixed 8k | 0.3377 | 0.4333 | 0.2766 | 0% | 0 | 69.3804 |
| packed_fixed 10k | 0.3038 | 0.3871 | 0.25 | 0% | 0 | 82.066 |
| packed_fixed 11k | 0.3953 | 0.4146 | 0.3778 | 0% | 0 | 208.0098 |
| packed_fixed 12k | 0.3516 | 0.3721 | 0.3333 | 0% | 0 | 76.0277 |
| packed_fixed 13k | 0.3158 | 0.48 | 0.2353 | 0% | 0 | 105.3601 |
| packed_fixed 15k | 0.3704 | 0.4688 | 0.3061 | 0% | 0 | 97.2876 |
| packed_fixed 17k | 0.3659 | 0.4286 | 0.3191 | 0% | 0 | 314.3545 |
| packed_fixed 20k | 0.3143 | 0.5238 | 0.2245 | 0% | 0 | 74.9157 |
| packed_fixed 23k | 0.3284 | 0.6471 | 0.22 | 0% | 0 | 109.5701 |
| packed_fixed 25k | 0.4096 | 0.5484 | 0.3269 | 0% | 0 | 211.4634 |
| packed_fixed 30k | 0.359 | 0.4516 | 0.2979 | 0% | 0 | 241.9714 |
| packed_fixed 40k | 0.3429 | 0.5455 | 0.25 | 0% | 0 | 171.6266 |
Dataset: repos/verademo
Key Metrics:
packed_fixed 13k at 0.3167packed_fixed 40k at 0.6316packed_fixed 1k at 0.4286project at 0% (0 per 100k lines)packed_fixed 12k at 95.7653s| Test Configuration | F1 | Precision | Recall | Skipped (%) | Skipped Lines / 100k | Processing Time (s) |
|---|---|---|---|---|---|---|
| project | 0.3025 | 0.3913 | 0.2466 | 0% | 0 | 215.7322 |
| file | 0.292 | 0.2857 | 0.2985 | 0% | 0 | 187.5239 |
| original 0.5k | 0.2857 | 0.241 | 0.3509 | 0% | 0 | 196.469 |
| original 0.7k | 0.3134 | 0.28 | 0.3559 | 0% | 0 | 186.2568 |
| original 1k | 0.292 | 0.2564 | 0.339 | 0% | 0 | 178.7028 |
| original 1.3k | 0.2917 | 0.25 | 0.35 | 0% | 0 | 179.4131 |
| original 1.6k | 0.2714 | 0.2375 | 0.3167 | 0% | 0 | 172.2454 |
| syntactic 1k | 0.25 | 0.2 | 0.3333 | 0% | 0 | 384.8079 |
| syntactic 3k | 0.2667 | 0.2128 | 0.3571 | 0% | 0 | 230.0782 |
| syntactic 5k | 0.2452 | 0.2088 | 0.2969 | 0% | 0 | 236.3227 |
| syntactic 6k | 0.2568 | 0.2184 | 0.3115 | 0% | 0 | 219.9031 |
| syntactic 7k | 0.2685 | 0.2273 | 0.3279 | 0% | 0 | 217.4358 |
| syntactic 10k | 0.2778 | 0.2273 | 0.3571 | 0% | 0 | 231.7776 |
| syntactic 13k | 0.2649 | 0.2273 | 0.3175 | 0% | 0 | 221.3313 |
| syntactic 15k | 0.2468 | 0.2065 | 0.3065 | 0% | 0 | 221.5634 |
| syntactic 17k | 0.2703 | 0.2353 | 0.3175 | 0% | 0 | 216.9881 |
| fixed_token 1k | 0.2286 | 0.1639 | 0.3774 | 0% | 0 | 290.0051 |
| fixed_token 3k | 0.242 | 0.1919 | 0.3276 | 0% | 0 | 209.3537 |
| fixed_token 5k | 0.2857 | 0.2632 | 0.3125 | 0% | 0 | 188.5324 |
| fixed_token 5.5k | 0.2533 | 0.2184 | 0.3016 | 0% | 0 | 170.7071 |
| fixed_token 10k | 0.3111 | 0.2692 | 0.3684 | 0% | 0 | 184.7888 |
| fixed_token 13k | 0.2985 | 0.2899 | 0.3077 | 0% | 0 | 183.603 |
| fixed_token 20k | 0.2837 | 0.274 | 0.2941 | 0% | 0 | 177.4314 |
| fixed_token 100k | 0.2837 | 0.274 | 0.2941 | 0% | 0 | 176.5943 |
| packed 1k | 0.2331 | 0.1776 | 0.3393 | 0% | 0 | 243.8184 |
| packed 3k | 0.259 | 0.2338 | 0.2903 | 0% | 0 | 166.0209 |
| packed 4k | 0.2791 | 0.2769 | 0.2812 | 0% | 0 | 131.6401 |
| packed 5.5k | 0.2946 | 0.25 | 0.3585 | 0% | 0 | 154.9145 |
| packed 6k | 0.3158 | 0.2692 | 0.3818 | 0% | 0 | 547.8309 |
| packed 6.5k | 0.2764 | 0.2881 | 0.2656 | 0% | 0 | 138.2367 |
| packed 7k | 0.2879 | 0.2969 | 0.2794 | 0% | 0 | 162.5977 |
| packed 10k | 0.2835 | 0.3529 | 0.2368 | 10.6521% | 0.0077 | 314.7808 |
| packed 11k | 0.2411 | 0.3617 | 0.1809 | 0% | 0 | 157.9426 |
| packed 12k | 0.2759 | 0.4324 | 0.2025 | 0% | 0 | 139.4706 |
| packed 13k | 0.2479 | 0.5357 | 0.1613 | 0% | 0 | 137.6573 |
| packed 15k | 0.3065 | 0.3455 | 0.2754 | 0% | 0 | 160.0651 |
| packed 17k | 0.2807 | 0.4444 | 0.2051 | 0% | 0 | 211.197 |
| packed 20k | 0.2656 | 0.5312 | 0.1771 | 0% | 0 | 129.8763 |
| packed 23k | 0.25 | 0.3404 | 0.1975 | 0% | 0 | 304.0643 |
| packed 25k | 0.2909 | 0.3636 | 0.2424 | 0% | 0 | 126.7201 |
| packed 30k | 0.2982 | 0.4722 | 0.2179 | 0% | 0 | 157.5058 |
| packed 40k | 0.2667 | 0.5714 | 0.1739 | 0% | 0 | 205.3469 |
| packed_fixed 1k | 0.2545 | 0.181 | 0.4286 | 0% | 0 | 300.2054 |
| packed_fixed 3k | 0.2781 | 0.2188 | 0.3818 | 0% | 0 | 193.5098 |
| packed_fixed 4k | 0.3089 | 0.2754 | 0.3519 | 0% | 0 | 141.1631 |
| packed_fixed 5.5k | 0.2815 | 0.2639 | 0.3016 | 0% | 0 | 139.0698 |
| packed_fixed 5k | 0.3016 | 0.2639 | 0.3519 | 0% | 0 | 155.5866 |
| packed_fixed 5.5k | 0.2992 | 0.2714 | 0.3333 | 0% | 0 | 128.6895 |
| packed_fixed 6k | 0.2794 | 0.2436 | 0.3276 | 0% | 0 | 153.9466 |
| packed_fixed 7k | 0.25 | 0.2329 | 0.2698 | 0% | 0 | 154.5024 |
| packed_fixed 8k | 0.3 | 0.2687 | 0.3396 | 0% | 0 | 119.982 |
| packed_fixed 10k | 0.2836 | 0.2879 | 0.2794 | 0% | 0 | 153.8308 |
| packed_fixed 11k | 0.3077 | 0.303 | 0.3125 | 0% | 0 | 114.3903 |
| packed_fixed 12k | 0.2667 | 0.4186 | 0.1957 | 0% | 0 | 95.7653 |
| packed_fixed 13k | 0.3167 | 0.3519 | 0.2879 | 0% | 0 | 161.4195 |
| packed_fixed 15k | 0.3051 | 0.3462 | 0.2727 | 0% | 0 | 122.0515 |
| packed_fixed 17k | 0.2769 | 0.2951 | 0.2609 | 0% | 0 | 222.4418 |
| packed_fixed 20k | 0.2636 | 0.3208 | 0.2237 | 0% | 0 | 234.2021 |
| packed_fixed 23k | 0.2302 | 0.4848 | 0.1509 | 0% | 0 | 137.6208 |
| packed_fixed 25k | 0.2286 | 0.4103 | 0.1584 | 0% | 0 | 222.494 |
| packed_fixed 30k | 0.3103 | 0.5143 | 0.2222 | 0% | 0 | 178.7284 |
| packed_fixed 40k | 0.1832 | 0.6316 | 0.1071 | 0% | 0 | 145.8342 |
Dataset: generated/*
Key Metrics:
packed 3k at 0.8packed 3k at 0.6667project at 1project at 0 per 100k linesoriginal 0.7k at 16.1603s| Test Configuration | F1 | Precision | Recall | Skipped (%) | Skipped Lines / 100k | Processing Time (s) |
|---|---|---|---|---|---|---|
| project | 0.7742 | 0.6316 | 1 | 0% | 0 | 17.6178 |
| file | 0.7059 | 0.5455 | 1 | 0% | 0 | 17.7077 |
| original 0.5k | 0.7333 | 0.6471 | 0.8462 | 0% | 0 | 16.2812 |
| original 0.7k | 0.7097 | 0.6111 | 0.8462 | 0% | 0 | 16.1603 |
| original 1k | 0.7273 | 0.6 | 0.9231 | 0% | 0 | 16.7223 |
| original 1.3k | 0.7273 | 0.6 | 0.9231 | 0% | 0 | 16.3634 |
| original 1.6k | 0.7742 | 0.6316 | 1 | 0% | 0 | 17.1588 |
| syntactic 1k | 0.6486 | 0.48 | 1 | 0% | 0 | 23.3573 |
| syntactic 3k | 0.6 | 0.4286 | 1 | 0% | 0 | 22.6888 |
| syntactic 5k | 0.6486 | 0.48 | 1 | 0% | 0 | 23.3752 |
| syntactic 6k | 0.6486 | 0.48 | 1 | 0% | 0 | 23.8655 |
| syntactic 7k | 0.6667 | 0.5 | 1 | 0% | 0 | 24.3255 |
| syntactic 10k | 0.6486 | 0.48 | 1 | 0% | 0 | 23.8538 |
| syntactic 13k | 0.6857 | 0.5217 | 1 | 0% | 0 | 22.9982 |
| syntactic 15k | 0.6667 | 0.5 | 1 | 0% | 0 | 24.1025 |
| syntactic 17k | 0.6667 | 0.5 | 1 | 0% | 0 | 22.7068 |
| fixed_token 1k | 0.6316 | 0.4615 | 1 | 0% | 0 | 18.1464 |
| fixed_token 3k | 0.7059 | 0.5455 | 1 | 0% | 0 | 17.4648 |
| fixed_token 5k | 0.7059 | 0.5455 | 1 | 0% | 0 | 18.1133 |
| fixed_token 5.5k | 0.7059 | 0.5455 | 1 | 0% | 0 | 17.601 |
| fixed_token 10k | 0.7059 | 0.5455 | 1 | 0% | 0 | 17.0174 |
| fixed_token 13k | 0.7059 | 0.5455 | 1 | 0% | 0 | 17.159 |
| fixed_token 20k | 0.7059 | 0.5455 | 1 | 0% | 0 | 17.3777 |
| fixed_token 100k | 0.7059 | 0.5455 | 1 | 0% | 0 | 17.7115 |
| packed 1k | 0.75 | 0.6 | 1 | 0% | 0 | 24.9797 |
| packed 3k | 0.8 | 0.6667 | 1 | 0% | 0 | 25.109 |
| packed 4k | 0.7742 | 0.6316 | 1 | 0% | 0 | 25.0905 |
| packed 5.5k | 0.7059 | 0.5455 | 1 | 0% | 0 | 26.6217 |
| packed 6k | 0.7273 | 0.5714 | 1 | 0% | 0 | 27.0882 |
| packed 6.5k | 0.7742 | 0.6316 | 1 | 0% | 0 | 25.6168 |
| packed 7k | 0.75 | 0.6 | 1 | 0% | 0 | 26.9844 |
| packed 10k | 0.7742 | 0.6316 | 1 | 0% | 0 | 25.2337 |
| packed 11k | 0.7059 | 0.5455 | 1 | 0% | 0 | 29.8796 |
| packed 12k | 0.7273 | 0.5714 | 1 | 0% | 0 | 29.312 |
| packed 13k | 0.75 | 0.6 | 1 | 0% | 0 | 25.7016 |
| packed 15k | 0.75 | 0.6 | 1 | 0% | 0 | 25.8391 |
| packed 17k | 0.75 | 0.6 | 1 | 0% | 0 | 25.7457 |
| packed 20k | 0.7742 | 0.6316 | 1 | 0% | 0 | 23.9379 |
| packed 23k | 0.6857 | 0.5217 | 1 | 0% | 0 | 31.5767 |
| packed 25k | 0.7273 | 0.5714 | 1 | 0% | 0 | 24.9255 |
| packed 30k | 0.7742 | 0.6316 | 1 | 0% | 0 | 26.0146 |
| packed 40k | 0.7742 | 0.6316 | 1 | 0% | 0 | 24.8103 |
| packed_fixed 1k | 0.6857 | 0.5217 | 1 | 0% | 0 | 17.566 |
| packed_fixed 3k | 0.7097 | 0.6111 | 0.8462 | 0% | 0 | 16.7408 |
| packed_fixed 4k | 0.6875 | 0.5789 | 0.8462 | 0% | 0 | 17.1787 |
| packed_fixed 5.5k | 0.7273 | 0.6 | 0.9231 | 0% | 0 | 27.2126 |
| packed_fixed 5k | 0.7273 | 0.6 | 0.9231 | 0% | 0 | 18.4112 |
| packed_fixed 5.5k | 0.6667 | 0.55 | 0.8462 | 0% | 0 | 17.0223 |
| packed_fixed 6k | 0.6875 | 0.5789 | 0.8462 | 0% | 0 | 16.8138 |
| packed_fixed 7k | 0.7333 | 0.6471 | 0.8462 | 0% | 0 | 16.6973 |
| packed_fixed 8k | 0.75 | 0.6 | 1 | 0% | 0 | 17.6294 |
| packed_fixed 10k | 0.75 | 0.6316 | 0.9231 | 0% | 0 | 17.881 |
| packed_fixed 11k | 0.7059 | 0.5714 | 0.9231 | 0% | 0 | 29.9918 |
| packed_fixed 12k | 0.75 | 0.6316 | 0.9231 | 0% | 0 | 26.662 |
| packed_fixed 13k | 0.7333 | 0.6471 | 0.8462 | 0% | 0 | 17.4988 |
| packed_fixed 15k | 0.7059 | 0.5714 | 0.9231 | 0% | 0 | 18.305 |
| packed_fixed 17k | 0.75 | 0.6 | 1 | 0% | 0 | 19.2815 |
| packed_fixed 20k | 0.6667 | 0.55 | 0.8462 | 0% | 0 | 18.943 |
| packed_fixed 23k | 0.75 | 0.6316 | 0.9231 | 0% | 0 | 28.3152 |
| packed_fixed 25k | 0.7273 | 0.5714 | 1 | 0% | 0 | 19.1012 |
| packed_fixed 30k | 0.7059 | 0.5714 | 0.9231 | 0% | 0 | 18.4352 |
| packed_fixed 40k | 0.7059 | 0.5714 | 0.9231 | 0% | 0 | 18.642 |