Skip to content

Refactor and enhance issue indexer to support both searching, filtering and paging #26012

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 108 commits into from
Jul 31, 2023
Merged
Changes from 1 commit
Commits
Show all changes
108 commits
Select commit Hold shift + click to select a range
7ae24a1
feat: IndexerMetadata
wolfogre Jun 16, 2023
1a13ce1
fix: use unique queue
wolfogre Jun 16, 2023
836ccf0
feat: query IndexerData
wolfogre Jun 16, 2023
d5c91d3
fix: new models
wolfogre Jun 16, 2023
3a2650f
feat: field for sorting
wolfogre Jul 5, 2023
9f464ed
fix: remove IsPublicRepo
wolfogre Jul 12, 2023
b1bcb88
fix: sortby
wolfogre Jul 12, 2023
ed16cd3
feat: add mapping
wolfogre Jul 12, 2023
84d6c0b
fix: build query
wolfogre Jul 13, 2023
29e3c7b
fix: label int64
wolfogre Jul 13, 2023
7a83ee9
feat: SearchIssues
wolfogre Jul 13, 2023
39c0769
test: use SearchIssues
wolfogre Jul 13, 2023
abdbfbe
fix: set default limit
wolfogre Jul 13, 2023
c6114f2
feat: RegisterFilterIssuesFunc
wolfogre Jul 13, 2023
f8f0b72
feat: WIP filterIssuesOfSearchResult
wolfogre Jul 13, 2023
668d831
feat: register db searching
wolfogre Jul 14, 2023
d547195
fix: db search
wolfogre Jul 14, 2023
b640caa
feat: db indexer
wolfogre Jul 14, 2023
da43c80
feat: db search
wolfogre Jul 14, 2023
0dcbb40
feat: ParsePaginator
wolfogre Jul 17, 2023
35eec57
fix: public repo
wolfogre Jul 17, 2023
a678836
feat: use SearchIssues
wolfogre Jul 17, 2023
840d7e3
fix: index issue
wolfogre Jul 17, 2023
2feead2
feat: getIssueIndexerData
wolfogre Jul 18, 2023
25eda69
feat: update option
wolfogre Jul 18, 2023
cff4ad6
feat: update issue indexer
wolfogre Jul 18, 2023
0a6c2d9
fix: docMapping
wolfogre Jul 18, 2023
fa35921
feat: bleve searching
wolfogre Jul 18, 2023
c97c8c2
chore: update bleve version
wolfogre Jul 18, 2023
f5e0f6c
fix: some fix
wolfogre Jul 18, 2023
0f1d898
fix: reuse SearchIssues in SearchIssuesByKeyword
wolfogre Jul 20, 2023
fad43d4
feat: web SearchIssues
wolfogre Jul 20, 2023
4c9af8b
fix: api ListIssues
wolfogre Jul 20, 2023
6953a68
feat: web ListIssues
wolfogre Jul 20, 2023
3af6404
fix: Paginator
wolfogre Jul 20, 2023
1df1ef4
feat: buildIssueOverview
wolfogre Jul 20, 2023
2b8aa89
fix: allIssueIDs
wolfogre Jul 20, 2023
4ca0660
feat: issues
wolfogre Jul 20, 2023
719173b
fix: meilisearch
wolfogre Jul 20, 2023
587c666
fix: applyLimit
wolfogre Jul 20, 2023
9bab931
fix: empty means SortByCreatedDesc
wolfogre Jul 20, 2023
86d8ee0
chore: remove SearchIssuesByKeyword
wolfogre Jul 20, 2023
4aa7fc7
chore: remove help comments
wolfogre Jul 20, 2023
e7e4edb
fix: lint code
wolfogre Jul 20, 2023
6473b31
test: fix case
wolfogre Jul 20, 2023
2759d46
test: fix case
wolfogre Jul 20, 2023
3c8cb1a
tests: new test framework for indexer
wolfogre Jul 20, 2023
dc8b61a
test: add comments
wolfogre Jul 20, 2023
3d00c2d
test: ExtraData
wolfogre Jul 20, 2023
5156771
test: tidy
wolfogre Jul 21, 2023
38c8024
test: empty
wolfogre Jul 21, 2023
0c8269e
fix: panic to read nil project
wolfogre Jul 21, 2023
c9d1399
fix: no label check
wolfogre Jul 21, 2023
6adf9d6
fix: meilisearch condition
wolfogre Jul 21, 2023
d18a68c
fix: db search
wolfogre Jul 21, 2023
e919d73
fix: db test
wolfogre Jul 21, 2023
6075007
fix: db searching
wolfogre Jul 21, 2023
535a691
fix: paginator
wolfogre Jul 21, 2023
4af28d6
fix: do default Paginator
wolfogre Jul 21, 2023
934c526
feat: use db if keyword is empty
wolfogre Jul 21, 2023
b7668aa
fix: applyLimit
wolfogre Jul 21, 2023
f6fec6b
test: cases for TestIndexer
wolfogre Jul 21, 2023
6a4999c
fix: LoadAttributes context
wolfogre Jul 21, 2023
f05b5b5
chore: lint code
wolfogre Jul 21, 2023
cb926ba
Merge branch 'main' into feature/enhance_issue_bleve_indexer
wolfogre Jul 24, 2023
05fb5f4
fix: filter by lables
wolfogre Jul 24, 2023
376a381
chore: add TBC
wolfogre Jul 24, 2023
8a64027
feat: IncludedAnyLabelIDs
wolfogre Jul 24, 2023
639a393
fix: use includedAnyLabels
wolfogre Jul 24, 2023
9618002
fix: load label names
wolfogre Jul 24, 2023
ee0fda4
Merge branch 'main' into feature/enhance_issue_bleve_indexer
wolfogre Jul 24, 2023
e840b6c
test: more cases
wolfogre Jul 24, 2023
28f1be6
fix: check IsAvailable
wolfogre Jul 24, 2023
e6383f3
test: more cases
wolfogre Jul 24, 2023
032363f
docs: add comments
wolfogre Jul 24, 2023
30a3375
debug: pq "gtestschema.label_issue" does not exist
wolfogre Jul 24, 2023
ddd5f2b
feat: improve enqueue
wolfogre Jul 25, 2023
37cf4c1
chore: use hotfix xorm
wolfogre Jul 25, 2023
ef27f6a
fix: es page size
wolfogre Jul 25, 2023
61b2317
Revert "chore: use hotfix xorm"
wolfogre Jul 26, 2023
fa03f68
Merge branch 'main' into feature/enhance_issue_bleve_indexer
wolfogre Jul 26, 2023
5797332
fix: fix unit test
wolfogre Jul 26, 2023
d4d540d
test: more cases
wolfogre Jul 26, 2023
f1b75ae
test: fix cases
wolfogre Jul 26, 2023
7369296
Merge branch 'main' into feature/enhance_issue_bleve_indexer
wolfogre Jul 26, 2023
0549ba3
feat: support elasticsearch
wolfogre Jul 26, 2023
4789728
feat: support es
wolfogre Jul 26, 2023
91ba549
feat: support meilisearch setting
wolfogre Jul 26, 2023
97e455f
test: waitData
wolfogre Jul 26, 2023
95e1f49
feat: settings for melisearch
wolfogre Jul 26, 2023
1e1328f
fix: accept es yellow
wolfogre Jul 27, 2023
bc7cb96
feat: support meilisearch
wolfogre Jul 27, 2023
da3a678
fix: RankingRules
wolfogre Jul 27, 2023
71d5ad2
chore: remove Imprecise
wolfogre Jul 27, 2023
c65718b
test: env for meilisearch
wolfogre Jul 27, 2023
439c798
chore: remove debug code
wolfogre Jul 27, 2023
70e780e
docs: add comments
wolfogre Jul 27, 2023
f12aba5
docs: add comments
wolfogre Jul 27, 2023
923b15f
chore: add meilisearch to hosts
wolfogre Jul 27, 2023
8f265d4
test: sleep when waiting
wolfogre Jul 27, 2023
8d386ce
Merge branch 'main' into feature/enhance_issue_bleve_indexer
wolfogre Jul 27, 2023
60638bb
Fix bug
lunny Jul 27, 2023
60fcb9c
Fix bug
lunny Jul 27, 2023
7010d53
fix: stats if zero found
wolfogre Jul 28, 2023
9473d81
Merge branch 'main' into feature/enhance_issue_bleve_indexer
wolfogre Jul 28, 2023
9b5f44f
Merge branch 'main' into feature/enhance_issue_bleve_indexer
GiteaBot Jul 31, 2023
dd71ef3
Merge branch 'main' into feature/enhance_issue_bleve_indexer
lunny Jul 31, 2023
9eeda00
Merge branch 'main' into feature/enhance_issue_bleve_indexer
techknowlogick Jul 31, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
feat: add mapping
  • Loading branch information
wolfogre committed Jul 20, 2023
commit ed16cd3216ec5f8f31b987166b3c4c353bb544e9
28 changes: 28 additions & 0 deletions modules/indexer/issues/bleve/bleve.go
Original file line number Diff line number Diff line change
Expand Up @@ -74,10 +74,38 @@ func generateIssueIndexMapping() (mapping.IndexMapping, error) {
textFieldMapping := bleve.NewTextFieldMapping()
textFieldMapping.Store = false
textFieldMapping.IncludeInAll = false

boolFieldMapping := bleve.NewBooleanFieldMapping()
boolFieldMapping.Store = false
boolFieldMapping.IncludeInAll = false

numberFieldMapping := bleve.NewNumericFieldMapping()
numberFieldMapping.Store = false
numberFieldMapping.IncludeInAll = false

docMapping.AddFieldMappingsAt("title", textFieldMapping)
docMapping.AddFieldMappingsAt("content", textFieldMapping)
docMapping.AddFieldMappingsAt("comments", textFieldMapping)

docMapping.AddFieldMappingsAt("is_pull", boolFieldMapping)
docMapping.AddFieldMappingsAt("is_closed", boolFieldMapping)
docMapping.AddFieldMappingsAt("labels", textFieldMapping)
docMapping.AddFieldMappingsAt("no_labels", boolFieldMapping)
docMapping.AddFieldMappingsAt("milestones", numberFieldMapping)
docMapping.AddFieldMappingsAt("no_milestones", boolFieldMapping)
docMapping.AddFieldMappingsAt("projects", numberFieldMapping)
docMapping.AddFieldMappingsAt("no_projects", boolFieldMapping)
docMapping.AddFieldMappingsAt("author", numberFieldMapping)
docMapping.AddFieldMappingsAt("assignee", numberFieldMapping)
docMapping.AddFieldMappingsAt("mentions", numberFieldMapping)
docMapping.AddFieldMappingsAt("reviewers", numberFieldMapping)
docMapping.AddFieldMappingsAt("requested_reviewers", numberFieldMapping)

docMapping.AddFieldMappingsAt("created_at", numberFieldMapping)
docMapping.AddFieldMappingsAt("updated_at", numberFieldMapping)
docMapping.AddFieldMappingsAt("closed_at", numberFieldMapping)
docMapping.AddFieldMappingsAt("due_date", numberFieldMapping)

if err := addUnicodeNormalizeTokenFilter(mapping); err != nil {
return nil, err
} else if err = mapping.AddCustomAnalyzer(issueIndexerAnalyzer, map[string]any{
Expand Down