| 123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136 |
- using Application.Abstractions.Data;
- using Microsoft.Extensions.DependencyInjection;
- using Microsoft.Extensions.Logging;
- using Microsoft.Extensions.Options;
- using SharedKernel;
- namespace Infrastructure.StockData;
- /// <summary>
- /// 거시경제지표 수집 (KOSIS 국가통계포털 statisticsData.do) — config Indicators 를 순회하며 각 지표의
- /// [today-BackfillYears, today] 시점 범위를 조회해 MacroIndicator upsert (UQ = Code+Period). 기본 07:50 KST 실행.
- /// 지표별 tblId/itmId/분류코드는 config(AppSettings.Kosis.Indicators)로 주도 — 코드 변경 없이 교정 가능하다.
- /// 콜 수가 적어(지표 6종 × 1회) quota 걱정은 낮지만 지표 사이 DelayMs 간격을 둔다. 빈 응답·err JSON·미고시 시점은
- /// 정상(0건)으로 취급해 그 지표만 skip 하고 다음으로 넘어간다. 예외 발생 시 경고 후 false 반환(다음 날/재시도).
- /// </summary>
- internal sealed class KosisSyncService(
- IServiceScopeFactory scopeFactory,
- IHttpClientFactory httpClientFactory,
- IOptions<AppSettings> settings,
- ILogger<KosisSyncService> logger
- ) : DailyScheduledService(logger)
- {
- protected override string JobName => "KosisSync";
- protected override TimeOnly TargetTime => ParseTime(settings.Value.Kosis.SyncTime, new TimeOnly(7, 50));
- protected override int MaxRetryCount => 2;
- protected override TimeSpan RetryDelay => TimeSpan.FromHours(1);
- protected override async Task<bool> RunOnceAsync(DateOnly todayKst, CancellationToken ct)
- {
- var cfg = settings.Value.Kosis;
- using var scope = scopeFactory.CreateScope();
- var collectorSettings = scope.ServiceProvider.GetRequiredService<ICollectorSettingsProvider>();
- if (!await collectorSettings.IsEnabledAsync(CollectorFlag.Kosis, ct))
- {
- return true;
- }
- cfg = cfg with { ApiKey = await collectorSettings.GetKeyAsync(CollectorKey.Kosis, ct) ?? cfg.ApiKey };
- var db = scope.ServiceProvider.GetRequiredService<IAppDbContext>();
- if (string.IsNullOrWhiteSpace(cfg.ApiKey))
- {
- Logger.LogWarning("[{Job}] Kosis:ApiKey 미설정 — 수집 skip", JobName);
- return true;
- }
- if (cfg.Indicators.Count == 0)
- {
- Logger.LogWarning("[{Job}] Kosis:Indicators 비어 있음 — 수집 skip", JobName);
- return true;
- }
- var client = httpClientFactory.CreateClient(KosisHttp.ClientName);
- var baseUrl = cfg.BaseUrl.TrimEnd('/');
- var years = cfg.BackfillYears > 0 ? cfg.BackfillYears : 5;
- var startYear = todayKst.AddYears(-years);
- var ok = true;
- foreach (var indicator in cfg.Indicators)
- {
- ct.ThrowIfCancellationRequested();
- if (string.IsNullOrWhiteSpace(indicator.Code) || string.IsNullOrWhiteSpace(indicator.OrgId) || string.IsNullOrWhiteSpace(indicator.TblId))
- {
- Logger.LogWarning("[{Job}] 지표 정의 불완전 (Code/OrgId/TblId) — skip: Code={Code}", JobName, indicator.Code);
- continue;
- }
- try
- {
- var (inserted, updated) = await FetchAndUpsertAsync(db, client, baseUrl, cfg.ApiKey, indicator, startYear, todayKst, ct);
- Logger.LogInformation("[{Job}] {Code}({Tbl}) 완료 — inserted={Ins}, updated={Upd}", JobName, indicator.Code, indicator.TblId, inserted, updated);
- }
- catch (OperationCanceledException)
- {
- throw;
- }
- catch (Exception ex)
- {
- ok = false;
- Logger.LogWarning(ex, "[{Job}] {Code}({Tbl}) 수집 실패 — 다음 지표 계속", JobName, indicator.Code, indicator.TblId);
- }
- if (cfg.DelayMs > 0)
- {
- await Task.Delay(cfg.DelayMs, ct);
- }
- }
- return ok;
- }
- private async Task<(int Inserted, int Updated)> FetchAndUpsertAsync(
- IAppDbContext db,
- HttpClient client,
- string baseUrl,
- string apiKey,
- AppSettings.KosisSection.KosisIndicator indicator,
- DateOnly startDate,
- DateOnly today,
- CancellationToken ct
- ) {
- var cycle = string.IsNullOrWhiteSpace(indicator.Cycle) ? "M" : indicator.Cycle.Trim().ToUpperInvariant();
- var startPrd = FormatPeriod(startDate, cycle);
- var endPrd = FormatPeriod(today, cycle);
- var itmId = string.IsNullOrWhiteSpace(indicator.ItmId) ? "ALL" : indicator.ItmId.Trim();
- var objL1 = string.IsNullOrWhiteSpace(indicator.ObjL1) ? "ALL" : indicator.ObjL1.Trim();
- var url = $"{baseUrl}?method=getList&apiKey={apiKey}&format=json&jsonVD=Y&orgId={indicator.OrgId}&tblId={indicator.TblId}&itmId={itmId}&objL1={objL1}&prdSe={cycle}&startPrdDe={startPrd}&endPrdDe={endPrd}";
- var json = await KosisHttp.GetStringWithRetryAsync(client, url, Logger, ct);
- var parsed = KosisParser.Parse(json);
- Logger.LogInformation("[{Job}] {Code} prdSe={Cycle} [{Start}~{End}] rows={Rows}", JobName, indicator.Code, cycle, startPrd, endPrd, parsed.Count);
- if (parsed.Count == 0)
- {
- return (0, 0);
- }
- return await KosisImport.UpsertAsync(db, indicator, parsed, ct);
- }
- /// <summary>KOSIS prdSe 별 시점 형식 — M=YYYYMM, Q=YYYYQ(분기 1~4), Y=YYYY.</summary>
- private static string FormatPeriod(DateOnly date, string cycle)
- {
- return cycle switch
- {
- "Y" => date.Year.ToString("D4"),
- "Q" => $"{date.Year:D4}{(date.Month - 1) / 3 + 1}",
- _ => $"{date.Year:D4}{date.Month:D2}"
- };
- }
- }
|