Merge pull request #10 from blackdice-cyber/main

main to dev

README.md

@@ -50,20 +50,27 @@ HYBRID_NS_ACCESS=andkWEzbBWRR2K6iw4edRvfd6MmNNjRx
 HYBRID_USER_AGENT=CSUX_blackdice
 ```
 
-6. To start the service, run:
+6. To start the service:
 
 ```
-sbin/gcf1 start
-Starting gcf1d services: ...succeeded
+cd /usr/local/gcf1/
+
+./sbin/gcf1 start
+
+# Starting gcf1d services: ...succeeded
 ```
 
 7. To download/update databases, run:
 
 ```
-bin/gcf1dbmng.sh etc urldb_download
-Success DB download .
-bin/gcf1dbmng.sh etc urldb_update
-Success DB update .
+cd /usr/local/gcf1/
+
+./bin/gcf1dbmng.sh etc urldb_download
+
+# Success DB download .
+
+./bin/gcf1dbmng.sh etc urldb_update
+# Success DB update .
 ```
 
 ## UDP Server Setup
@@ -85,14 +92,14 @@ npm install
 
 ```
 cp .env.example .env
-PORT=3000
+
+# --------- example contents ---------
+# UDP_PORT=33333
+# HTTP_PORT=3333
+# ------------------------------------
 ```
 
-## Running the Server
-
-To start the UDP server, use the following command:
-
-```
-npm start
-```
-This will run the server and bind it to the port specified in your .env file.
+## Digital Ocean Deployment details
+1. The Node server is installed to `/opt/netstar-categorizer`
+2. Rather than using `npm start` (which doesn't restart after a crash) the we use **[PM2](https://pm2.keymetrics.io/)**, which provides a more robust runtime
+3. The full command to start the server using pm2 is `pm2 start npm --name "netstar-categorizer" --cwd /opt/netstar-categorizer/ -- start` - for convenience, there's a `run-me.sh` script in `/root` that will execute this command. 

claude.md

@@ -0,0 +1,290 @@
+# Claude Development Guide - NetStar Categorizer
+
+## Project Overview
+
+**NetStar Categorizer** is a Node.js microservice that provides domain name/FQDN categorization using the inCompass NetStar SDK. It exposes both UDP and HTTP interfaces for real-time content classification with automatic daily database updates.
+
+### Core Purpose
+- Categorize domains into standardized categories using NetStar's content database
+- Map raw NetStar category IDs to organization-specific category codes
+- Provide both simple and detailed categorization results with reputation/age ratings
+- Maintain updated categorization databases through automated cron jobs
+
+## Key Architecture
+
+### Technology Stack
+- **Runtime**: Node.js v14+
+- **Web Framework**: Express.js v4
+- **External SDK**: inCompass NetStar v3.1.0-2 (C++ library for categorization)
+- **Containerization**: Docker + Kubernetes
+- **CI/CD**: CircleCI
+- **Infrastructure**: Cloud-agnostic (DigitalOcean, Google Cloud, etc.)
+
+### Core Services
+1. **HTTP Server** (Port 3333) - REST API for categorization requests
+2. **UDP Server** (Port 33333) - Legacy UDP interface for categorization
+3. **Cron Service** - Daily database updates via Kubernetes CronJob
+4. **Category Mapping** - NetStar ID → Organization Code conversion (singleton pattern)
+
+## Project Structure
+
+```
+src/
+├── server.js                              # Entry point: initializes UDP + HTTP servers
+├── app.js                                 # Core logic: orchestrates categorization flow
+├── client.js                              # UDP test client
+├── cron.js                                # Scheduled database update logic
+├── use-cases/
+│   ├── get-category-use-case.js          # Executes NetStar gcf1check command
+│   ├── category-converter-use-case.js    # Maps NetStar IDs → org codes (singleton)
+│   ├── parse-detailed-category-use-case.js # Parses detailed output with ratings
+│   └── update-categories-use-case.js     # Updates NetStar databases
+└── etc/
+    └── categories-mapping.json           # Mapping table: NetStar ID → Zvelo codes
+
+deployment/
+├── deployment.yaml                       # Kubernetes deployment manifest
+└── staging/deployment.yaml               # Staging-specific config
+
+.circleci/config.yml                      # CI/CD pipeline (builds, tests, deploys)
+Dockerfile                                # Container build specification
+package.json                              # Node dependencies + scripts
+makefile                                  # Convenience commands
+test-detailed.http                        # HTTP endpoint test file
+```
+
+## Language Standards
+
+### Comments and Error Messages - ENGLISH ONLY
+**All code comments, error messages, log statements, and documentation must be written in English.**
+
+This includes:
+- ✅ Code comments explaining logic
+- ✅ Error messages and exception messages
+- ✅ Console.log, console.error, and logging statements
+- ✅ Variable and function names
+- ✅ Commit messages
+- ✅ Code review feedback
+- ✅ Documentation strings (JSDoc, etc.)
+
+**Examples:**
+```javascript
+// Correct: English comment
+function mapCategoryId(netstarId) {
+  if (!netstarId) {
+    throw new Error('NetStar ID is required')
+  }
+  // Map to organization category code
+  return categoryConverter.convert(netstarId)
+}
+
+// Incorrect: Portuguese comment
+function mapCategoryId(netstarId) {
+  if (!netstarId) {
+    throw new Error('ID do NetStar é obrigatório')  // ❌ ERROR IN PORTUGUESE
+  }
+  // Mapear para código de categoria da organização  // ❌ COMMENT IN PORTUGUESE
+  return categoryConverter.convert(netstarId)
+}
+```
+
+## Coding Conventions
+
+### Code Style
+- Use **ES6 syntax** (const/let, arrow functions, template literals)
+- **No semicolons** in new code (already established pattern)
+- Functional/modular design - keep files focused on single responsibility
+- Use **singleton pattern** for shared state (see: `CategoryConverterUseCase`)
+- **All comments in English** - see Language Standards section
+
+### Use Case Pattern
+- Each business operation gets a dedicated use case class in `src/use-cases/`
+- Use case classes should have a clear, single responsibility
+- Example:
+  ```javascript
+  class GetCategoryUseCase {
+    async execute(fqdn) {
+      // implementation
+    }
+  }
+  ```
+
+### Environment Variables
+- Defined in `.env` (create from `.env.example`)
+- `UDP_PORT=33333` - UDP server listen port
+- `HTTP_PORT=3333` - HTTP server listen port
+
+## HTTP API Endpoints
+
+### `POST /` - Basic Categorization
+- **Input**: `{"fqdn": "example.com"}`
+- **Output**: `{"result": [10009, 10010]}` (array of category IDs)
+- **Use Case**: Quick lookups when detailed info not needed
+
+### `POST /detailed` - Detailed Categorization
+- **Input**: `{"fqdn": "example.com"}`
+- **Output**: Full category info with reputation score, age rating, primary/secondary categories, and human-readable names
+- **Use Case**: Comprehensive categorization for security decisions
+- **Includes**: `result` array matching the `/` endpoint format
+
+### `POST /full` - Complete Raw Categorization Output
+- **Input**: `{"fqdn": "example.com"}`
+- **Output**: Complete detailed structure with all 35 fields from NetStar including:
+  - Primary, secondary, and security categories (with IDs, names, and mapped IDs)
+  - Reputation score and name
+  - Matching flags and their descriptions
+  - Age rating score and name
+  - All 9 category group classifications (Internet/Infrastructure, Malware/Security, Dangerous/Harmful, Adult, Business/Government, Personal, Computing/Technology, Social Media, Miscellaneous)
+  - Volume index
+  - Submitted URL
+- **Use Case**: Complete diagnostic and analysis when all categorization data is needed
+- **Includes**: `result` array matching the `/` endpoint format
+
+### UDP Server (Port 33333)
+- **Input**: Raw domain string (e.g., `"example.com"`)
+- **Output**: JSON-formatted result (same as HTTP `/` endpoint)
+- **Legacy Interface**: Maintained for backward compatibility
+
+## Development Workflow
+
+### Setup & Running Locally
+```bash
+# Install dependencies
+npm install
+
+# Development with auto-reload
+npm run dev:server    # Watch mode for server changes
+npm run dev:client    # Run UDP client for testing
+
+# Production
+npm start             # Start both servers
+
+# NetStar service commands (Linux system)
+make gcf1-start       # Start NetStar service
+make gcf1-download    # Download category databases
+make gcf1-update      # Update category databases
+```
+
+### Testing APIs
+Use `test-detailed.http` in VS Code REST Client extension:
+1. Open the file
+2. Click "Send Request" on each endpoint
+3. View responses in the side panel
+
+### Git Workflow
+- **Main Branch**: `main` - production stable code
+- **Development Branch**: `development` - feature integration
+- **Feature Branches**: Create from `development`, merge back via PR
+- Recent commits show HTTP approach implementation and cron job additions
+- **Commit Messages**: Must be in English
+
+## Deployment
+
+### Docker
+- **Base Image**: Ubuntu 22.04
+- **Includes**: Boost libraries, Node.js, NetStar SDK
+- **Exposes**: Port 3000 (UDP)
+- Build: `docker build -t netstar-categorizer .`
+
+### Kubernetes (Production)
+- **Namespace**: `blackdice`
+- **Deployment**: Single replica in appropriate cluster
+- **CronJob**: Daily database updates at 00:00 UTC
+- **Ingress**: `netstar-cat-dev.blackdice.ai` (DNS varies by environment)
+- **Branches to Environments**:
+  - `development` → development cluster
+  - `qa` → QA cluster
+  - `staging` → staging cluster
+  - `production` → production cluster
+  - `gke-staging`, `gke-pov` → specific GKE clusters
+
+### CI/CD Pipeline (CircleCI)
+- Automatically builds Docker image on push
+- Tags image with commit SHA
+- Deploys to appropriate Kubernetes cluster based on branch
+- Release deployments via git tags
+
+## Key Technical Details
+
+### Category Mapping System
+- **Source**: NetStar SDK returns numeric category IDs (e.g., 101, 102)
+- **Mapping File**: `src/etc/categories-mapping.json`
+- **Target**: Maps to organization's Zvelo pattern codes (e.g., 10075, 10078)
+- **Singleton Implementation**: `CategoryConverterUseCase` maintains single instance across app
+- **Example Mapping**:
+  - NetStar 101 (Illegal Activities) → Zvelo 10075
+  - NetStar 201 (Terrorism/Extremists) → Zvelo 10018
+
+### NetStar SDK Integration
+- **Command**: `gcf1check` - queries the NetStar database for domain categorization
+- **Child Process**: Executed via Node.js `child_process` module
+- **Output Parsing**: Raw output parsed into JSON structure
+- **Detailed Mode**: Includes reputation scores and age ratings in output
+
+### Automatic Database Updates
+- **Mechanism**: Kubernetes CronJob at 0 0 * * * (daily at midnight UTC)
+- **Fallback**: Manual update via `make gcf1-update`
+- **Purpose**: Keeps categorization database current with latest NetStar classifications
+
+## Common Tasks
+
+### Adding a New Endpoint
+1. Create a corresponding use case in `src/use-cases/`
+2. Add route in `src/app.js` that calls the use case
+3. Export and test in `test-detailed.http`
+4. Update this guide if it's a significant feature
+5. Ensure all error messages and comments are in English
+
+### Updating Category Mappings
+1. Modify `src/etc/categories-mapping.json` with new ID mappings
+2. Restart the service (singleton will reload on next request)
+3. Test with both HTTP and UDP interfaces
+
+### Debugging
+- **Server Logs**: Check Docker/Kubernetes logs for errors
+- **Cron Logs**: View Kubernetes CronJob logs for database update issues
+- **UDP Testing**: Use `npm run dev:client` to test directly
+- **HTTP Testing**: Use `test-detailed.http` with VS Code REST Client
+- **Error Messages**: All error logs must be in English
+
+### Troubleshooting
+- **NetStar Service Not Running**: Run `make gcf1-start`
+- **Stale Categories**: Manually run `make gcf1-update` or wait for cron job
+- **Port Conflicts**: Ensure ports 3333 (HTTP) and 33333 (UDP) are available
+- **Docker Build Issues**: Check that Boost C++ libraries are installed correctly
+
+## Current Development Status
+
+### Recent Work
+- ✅ HTTP server implementation (alongside UDP)
+- ✅ Detailed categorization with reputation/age ratings
+- ✅ Cron job for automated daily updates
+- ✅ Singleton category converter pattern
+- 🔄 Work in Progress:
+  - `playground.js` - experimental/testing code
+  - `parse-detailed-category-use-case.js` - new detailed parsing feature
+  - Enhanced `server.js` - expanded server capabilities
+
+### Known Modified Files
+- `playground.js` - development/testing (can be cleaned up)
+- `src/server.js` - recent enhancements
+- `makefile` - new convenience commands
+- `test-detailed.http` - expanded test coverage
+
+## Guidelines for Contributions
+
+1. **Follow Existing Patterns**: Use use-case classes, follow module structure
+2. **Test Before Committing**: Use `test-detailed.http` for API changes
+3. **Update Mappings Properly**: Edit `categories-mapping.json`, not hardcode values
+4. **Document Breaking Changes**: Update this guide if architecture changes
+5. **Keep CircleCI Happy**: Ensure Docker build succeeds and K8s deployment configs are valid
+6. **Don't Skip Steps**: Always test UDP and HTTP interfaces for categorization changes
+7. **Language Standards**: All comments, error messages, and logs must be in English
+
+## Resources & External Documentation
+
+- **NetStar SDK**: Installed in Docker, documentation in inCompass SDK v3.1.0-2
+- **Express.js**: https://expressjs.com
+- **Node.js Child Process**: https://nodejs.org/api/child_process.html
+- **Kubernetes**: https://kubernetes.io/docs
+- **CircleCI**: Configuration at `.circleci/config.yml`

makefile

@@ -0,0 +1,8 @@
+gcf1-start:
+	cd /usr/local/gcf1 && sbin/gcf1 start
+	
+gcf1-download:
+	cd /usr/local/gcf1 && bin/gcf1dbmng.sh etc urldb_download
+
+gcf1-update:
+	cd /usr/local/gcf1 && bin/gcf1dbmng.sh etc urldb_update

playground.js

@@ -7,12 +7,12 @@ const { spawn, exec } = require("node:child_process")
 // })
 
 
-exec("echo '99999.incompass.netstar-inc.com' | bin/gcf1check.sh etc check_categorize_hybrid", { cwd: '/usr/local/gcf1' }, (error, stdout, stderr)=>{
+exec("echo 'li12.pages.dev' | bin/gcf1check.sh etc check_categorize_hybrid", { cwd: '/usr/local/gcf1' }, (error, stdout, stderr)=>{
   if (error) {
     console.error(error)
     return
   }
 
-  console.log(stdout.trim())
+    console.log(stdout.trim().split(/\t|\s{2,}/))
 
 }) 

src/app.js

@@ -7,10 +7,12 @@ const getCategory = new GetCategoryUseCase();
 const categoryConverter = new CategoryConverterUseCase({categoriesMapping});
 
 module.exports = async function app(domain){
+  console.log(domain)
   const category = await getCategory.execute(domain)
+  console.log(category)
   const categoryConverted = await categoryConverter.execute(category)
   console.log(categoryConverted)
-  return categoryConverted
+  return [categoryConverted]
 }
 
 

src/cron.js

@@ -3,8 +3,10 @@ const { UpdateCategoriesUseCase } = require('./use-cases/update-categories-use-c
 const updateCategories = new UpdateCategoriesUseCase()
 
 module.exports = async function cron() {
+  updateCategories.execute()
+
   const oneDay = 1000 * 60 * 60 * 24
-  setImmediate(() => {
+  setInterval(() => {
     updateCategories.execute()
   }, oneDay)
 }

src/etc/categories-mapping.json

@@ -1074,7 +1074,7 @@
       "id": "20097",
       "description": "VPN",
       "related": [
-          "10451"
+          "10460"
       ]
   },
   {
@@ -1109,7 +1109,7 @@
       "id": "20103",
       "description": "Code Repositories",
       "related": [
-          "10429"
+          "10047"
       ]
   },
   {
@@ -1462,4 +1462,4 @@
           "10094"
       ]
   }
-]
+]

src/server.js

@@ -5,14 +5,23 @@ const app = require('./app')
 const bodyParser = require('body-parser')
 const cron = require('./cron')
 const express = require('express')
+const { ParseDetailedCategoryUseCase } = require('./use-cases/parse-detailed-category-use-case')
+const { ParseFullCategoryUseCase } = require('./use-cases/parse-full-category-use-case')
+const { CategoryConverterUseCase } = require('./use-cases/category-converter-use-case')
+const categoriesMapping = require('./etc/categories-mapping.json')
 
 const httpServer = express()
 httpServer.use(bodyParser.json()) // for parsing application/json
 httpServer.use(bodyParser.urlencoded({ extended: true })) // for parsing application/x-www-form-urlencoded
 cron()
 
-const UDP_PORT = process.env.UDP_PORT 
-const HTTP_PORT = process.env.HTTP_PORT 
+const UDP_PORT = process.env.UDP_PORT
+const HTTP_PORT = process.env.HTTP_PORT
+
+// Initialize use cases
+const categoryConverter = new CategoryConverterUseCase({ categoriesMapping })
+const parseDetailedCategory = new ParseDetailedCategoryUseCase({ categoryConverter })
+const parseFullCategory = new ParseFullCategoryUseCase({ categoryConverter }) 
 
 const server = dgram.createSocket('udp4');
 
@@ -50,27 +59,55 @@ server.on('listening', () => {
 server.bind(UDP_PORT);
 
 httpServer.post('/', async (req, res) => {
+  console.log("new request")
   try {
     const { fqdn } = req.body
 
     const categories = await app(fqdn)
-  
+
     let result = {}
-  
-    if (categories) 
+
+    if (categories)
     result = {
       result: categories
     }
-  
+
     res.status(200).json(JSON.stringify(result))
   } catch (err) {
     res.status(500).json(err)
   }
-  
+
+})
+
+httpServer.post('/detailed', async (req, res) => {
+  try {
+    const { fqdn } = req.body
+
+    const detailedResult = await parseDetailedCategory.execute(fqdn)
+
+    res.status(200).json(detailedResult)
+  } catch (err) {
+    console.error('Error in /detailed endpoint:', err)
+    res.status(500).json({ error: err.message || err })
+  }
+})
+
+httpServer.post('/full', async (req, res) => {
+  try {
+    const { fqdn } = req.body
+
+    const fullResult = await parseFullCategory.execute(fqdn)
+
+    res.status(200).json(fullResult)
+  } catch (err) {
+    console.error('Error in /full endpoint:', err)
+    res.status(500).json({ error: err.message || err })
+  }
 })
 
 httpServer.listen(HTTP_PORT, () => {
   console.log('HTTP server listening 3333')
+  console.log('UDP server listening 3334')
 })
 
 // Response

src/test-app.js

@@ -0,0 +1,10 @@
+const app = require('./app');
+
+(async () => {
+  try {
+    const categories = await app('www.facebook.com');
+    console.log(categories);
+  } catch (error) {
+    console.error('Error occurred:', error);
+  }
+})();

src/use-cases/category-converter-use-case.js

@@ -11,7 +11,7 @@ class CategoryConverterUseCase {
   execute(category) {
     const entry = this.categoriesMapping.find(item => item.id === category);
 
-    return entry ? entry.related[0].split(', ').map(str => str.trim()) : null;
+    return entry ? entry.related[0] : null;
   }
 }
 

src/use-cases/get-category-use-case.js

@@ -10,11 +10,11 @@ class GetCategoryUseCase {
           return;
         }
 
-        const outputParts = stdout.split(/\s+/);
-        if (outputParts[3]) {
-          const categoryId = outputParts[2];
-          console.log({ categoryId });
-          resolve(categoryId);
+        const outputParts = stdout.split(/\t/);
+        if (outputParts[6] !== '0') {
+          resolve(outputParts[6])
+        } else if (outputParts[3]) {
+          resolve(outputParts[2]);
         } else {
           console.log({ error: 'Category ID not found' });
           reject('Category ID not found');

src/use-cases/parse-detailed-category-use-case.js

@@ -0,0 +1,101 @@
+const { exec } = require("node:child_process")
+
+class ParseDetailedCategoryUseCase {
+  constructor({ categoryConverter }) {
+    this.categoryConverter = categoryConverter;
+  }
+
+  execute(domain) {
+    return new Promise((resolve, reject) => {
+      exec(`echo ${domain} | bin/gcf1check.sh etc check_categorize_hybrid`,
+           { cwd: '/usr/local/gcf1' },
+           (error, stdout, stderr) => {
+             if (error) {
+               console.error(error);
+               reject(error);
+               return;
+             }
+
+             try {
+               const parsed = this.parseOutput(stdout);
+               resolve(parsed);
+             } catch (parseError) {
+               console.error('Parse error:', parseError);
+               reject(parseError);
+             }
+           });
+    });
+  }
+
+  parseOutput(output) {
+    // Extract all quoted strings in order
+    const quotedStrings = [];
+    const quoteRegex = /"([^"]*)"/g;
+    let match;
+    while ((match = quoteRegex.exec(output)) !== null) {
+      quotedStrings.push(match[1]);
+    }
+
+    // Split on tabs (primary separator in NetStar output)
+    const parts = output.trim().split('\t');
+    let quotedIndex = 0;
+
+    // Position mapping based on NetStar output structure:
+    // parts[0]: Categorized
+    // parts[1]: Matching Index
+    // parts[2]: Primary Category ID
+    // parts[3]: Primary Category Name (quoted)
+    // parts[4]: Secondary Category ID
+    // parts[5]: Secondary Category Name (quoted if secondary exists)
+    // parts[6]: Security Category ID
+    // parts[7]: Security Category Name
+    // parts[8]: Reputation Score ID
+    // parts[9]: Reputation Score Name (quoted)
+    // parts[10]: Matching Flag Value
+    // parts[11]: Matching Flag Names
+    // parts[12]: Age Rating ID
+    // parts[13]: Age Rating Name (quoted)
+
+    const primaryId = parseInt(parts[2]);
+    const primaryName = quotedStrings[quotedIndex++];
+    const primaryMapped = this.categoryConverter.execute(String(primaryId));
+
+    const secondaryId = parts[4] !== '0' && parts[4] !== '-' ? parseInt(parts[4]) : null;
+    let secondaryName = null;
+    let secondaryMapped = null;
+
+    if (secondaryId !== null) {
+      secondaryName = quotedStrings[quotedIndex++];
+      secondaryMapped = this.categoryConverter.execute(String(secondaryId));
+    } else {
+      secondaryName = parts[5] === '-' ? null : parts[5];
+    }
+
+    const reputationId = parseInt(parts[8]);
+    const reputationName = quotedStrings[quotedIndex++];
+
+    const ageRatingId = parseInt(parts[12]);
+    const ageRatingName = quotedStrings[quotedIndex++];
+
+    // Build result array in the same format as the / endpoint
+    const resultArray = [String(primaryMapped)];
+    if (secondaryMapped !== null) {
+      resultArray.push(String(secondaryMapped));
+    }
+
+    return {
+      result: resultArray,
+      primary: primaryMapped,
+      primary_name: primaryName,
+      secondary: secondaryMapped,
+      secondary_name: secondaryName,
+      reputation: reputationId,
+      reputation_name: reputationName,
+      age_rating: ageRatingId,
+      age_rating_name: ageRatingName,
+      raw_output: output.trim()
+    };
+  }
+}
+
+module.exports = { ParseDetailedCategoryUseCase }

src/use-cases/parse-full-category-use-case.js

@@ -0,0 +1,172 @@
+const { exec } = require("node:child_process")
+
+class ParseFullCategoryUseCase {
+  constructor({ categoryConverter }) {
+    this.categoryConverter = categoryConverter;
+  }
+
+  execute(domain) {
+    return new Promise((resolve, reject) => {
+      exec(`echo ${domain} | bin/gcf1check.sh etc check_categorize_hybrid`,
+           { cwd: '/usr/local/gcf1' },
+           (error, stdout, stderr) => {
+             if (error) {
+               console.error(error);
+               reject(error);
+               return;
+             }
+
+             try {
+               const parsed = this.parseOutput(stdout);
+               resolve(parsed);
+             } catch (parseError) {
+               console.error('Parse error:', parseError);
+               reject(parseError);
+             }
+           });
+    });
+  }
+
+  parseOutput(output) {
+    const quotedStrings = [];
+    const quoteRegex = /"([^"]*)"/g;
+    let match;
+    while ((match = quoteRegex.exec(output)) !== null) {
+      quotedStrings.push(match[1]);
+    }
+
+    const parts = output.trim().split('\t');
+    let quotedIndex = 0;
+
+    // Extract primary category (parts[2], parts[3])
+    const primaryId = parseInt(parts[2]);
+    const primaryName = quotedStrings[quotedIndex++];
+    const primaryMapped = this.categoryConverter.execute(String(primaryId));
+
+    // Extract secondary category (parts[4], parts[5])
+    const secondaryId = parts[4] !== '0' && parts[4] !== '-' ? parseInt(parts[4]) : null;
+    let secondaryName = null;
+    let secondaryMapped = null;
+    if (secondaryId !== null && parts[5] !== '-') {
+      secondaryName = quotedStrings[quotedIndex++];
+      secondaryMapped = this.categoryConverter.execute(String(secondaryId));
+    } else {
+      secondaryName = parts[5] === '-' ? null : parts[5];
+    }
+
+    // Extract security category (parts[6], parts[7])
+    const securityId = parts[6] !== '0' && parts[6] !== '-' ? parseInt(parts[6]) : null;
+    let securityName = null;
+    let securityMapped = null;
+    if (securityId !== null && parts[7] !== '-') {
+      securityName = quotedStrings[quotedIndex++];
+      securityMapped = this.categoryConverter.execute(String(securityId));
+    } else {
+      securityName = parts[7] === '-' ? null : parts[7];
+    }
+
+    // Extract reputation (parts[8], parts[9])
+    const reputationId = parseInt(parts[8]);
+    const reputationName = quotedStrings[quotedIndex++];
+
+    // Extract matching flag (parts[10], parts[11])
+    const matchingFlagValue = parts[10];
+    const matchingFlagNames = parts[11] ? parts[11].split('|') : [];
+
+    // Extract age rating (parts[12], parts[13])
+    const ageRatingId = parseInt(parts[12]);
+    const ageRatingName = quotedStrings[quotedIndex++];
+
+    // Extract category groups starting from parts[16] (after 2 empty fields)
+    // Each group has: id (number), name (quoted string)
+    // Pattern: 0 "Internet/Infrastructure" 0 "Malware/Security" ...
+    const categoryGroups = {
+      internet_infrastructure: {
+        id: parseInt(parts[16]) || 0,
+        name: quotedStrings[quotedIndex++]
+      },
+      malware_security: {
+        id: parseInt(parts[18]) || 0,
+        name: quotedStrings[quotedIndex++]
+      },
+      dangerous_harmful: {
+        id: parseInt(parts[20]) || 0,
+        name: quotedStrings[quotedIndex++]
+      },
+      adult: {
+        id: parseInt(parts[22]) || 0,
+        name: quotedStrings[quotedIndex++]
+      },
+      business_government: {
+        id: parseInt(parts[24]) || 0,
+        name: quotedStrings[quotedIndex++]
+      },
+      personal: {
+        id: parseInt(parts[26]) || 0,
+        name: quotedStrings[quotedIndex++]
+      },
+      computing_technology: {
+        id: parseInt(parts[28]) || 0,
+        name: quotedStrings[quotedIndex++]
+      },
+      social_media: {
+        id: parseInt(parts[30]) || 0,
+        name: quotedStrings[quotedIndex++]
+      },
+      miscellaneous: {
+        id: parseInt(parts[32]) || 0,
+        name: quotedStrings[quotedIndex++]
+      }
+    };
+
+    // Extract volume index and submitted URL
+    const volumeIndex = parts[35];
+    const submittedUrl = parts[36];
+
+    const result = {
+      result_status: parts[0],
+      matching_index: parseInt(parts[1]),
+      primary_category: {
+        id: primaryId,
+        name: primaryName,
+        mapped_id: String(primaryMapped)
+      },
+      secondary_category: {
+        id: secondaryId,
+        name: secondaryName,
+        mapped_id: secondaryMapped ? String(secondaryMapped) : null
+      },
+      security_category: {
+        id: securityId,
+        name: securityName,
+        mapped_id: securityMapped ? String(securityMapped) : null
+      },
+      reputation: {
+        id: reputationId,
+        name: reputationName
+      },
+      matching_flag: {
+        value: matchingFlagValue,
+        names: matchingFlagNames
+      },
+      age_rating: {
+        id: ageRatingId,
+        name: ageRatingName
+      },
+      category_groups: categoryGroups,
+      volume_index: volumeIndex,
+      submitted_url: submittedUrl
+    };
+
+    // Add result array in the same format as / endpoint
+    const resultArray = [result.primary_category.mapped_id];
+    if (result.secondary_category.mapped_id !== null) {
+      resultArray.push(result.secondary_category.mapped_id);
+    }
+    result.result = resultArray;
+
+    return result;
+  }
+}
+
+module.exports = { ParseFullCategoryUseCase }

src/use-cases/update-categories-use-case.js

@@ -1,20 +1,26 @@
-const { exec } = require("node:child_process")
+const { exec } = require("node:child_process");
 
 class UpdateCategoriesUseCase {
   execute() {
-    return new Promise((_, reject) => {
-      exec(`bin/gcf1dbmng.sh etc urldb_update`, { cwd: '/usr/local/gcf1' }, (error, stdout, stderr) => {
-        if (error) {
-          console.error(error);
-          reject(error);
+    exec(`bin/gcf1dbmng.sh etc urldb_download`, { cwd: '/usr/local/gcf1' }, (error, stdout, stderr) => {
+      if (error) {
+        console.error('Erro no primeiro comando:', error);
+        return;
+      }
+
+      console.log('Saída do primeiro comando:', stdout);
+
+      // Executa o segundo comando após o primeiro ter sido concluído
+      exec(`bin/gcf1dbmng.sh etc urldb_update`, { cwd: '/usr/local/gcf1' }, (error2, stdout2, stderr2) => {
+        if (error2) {
+          console.error('Erro no segundo comando:', error2);
           return;
         }
 
-        console.log(stdout)
+        console.log('Saída do segundo comando:', stdout2);
       });
     });
   }
 }
 
-module.exports = { UpdateCategoriesUseCase }
-
+module.exports = { UpdateCategoriesUseCase };

test-detailed.http

@@ -0,0 +1,188 @@
+@baseUrl = http://localhost:3333
+
+# ============================================
+# POST / - Basic Categorization Endpoint
+# Returns: Simple result array of category IDs
+# ============================================
+
+### Basic endpoint - Facebook
+POST {{baseUrl}}/
+Content-Type: application/json
+
+{
+  "fqdn": "facebook.com"
+}
+
+### Basic endpoint - Google
+POST {{baseUrl}}/
+Content-Type: application/json
+
+{
+  "fqdn": "google.com"
+}
+
+### Basic endpoint - TikTok
+POST {{baseUrl}}/
+Content-Type: application/json
+
+{
+  "fqdn": "tiktok.com"
+}
+
+### Basic endpoint - YouTube
+POST {{baseUrl}}/
+Content-Type: application/json
+
+{
+  "fqdn": "youtube.com"
+}
+
+### Basic endpoint - Reddit
+POST {{baseUrl}}/
+Content-Type: application/json
+
+{
+  "fqdn": "reddit.com"
+}
+
+### Basic endpoint - Twitter
+POST {{baseUrl}}/
+Content-Type: application/json
+
+{
+  "fqdn": "twitter.com"
+}
+
+# ============================================
+# POST /detailed - Detailed Categorization
+# Returns: Categories with names, reputation, age rating
+# ============================================
+
+### Detailed endpoint - Facebook
+POST {{baseUrl}}/detailed
+Content-Type: application/json
+
+{
+  "fqdn": "facebook.com"
+}
+
+### Detailed endpoint - Google
+POST {{baseUrl}}/detailed
+Content-Type: application/json
+
+{
+  "fqdn": "google.com"
+}
+
+### Detailed endpoint - TikTok
+POST {{baseUrl}}/detailed
+Content-Type: application/json
+
+{
+  "fqdn": "tiktok.com"
+}
+
+### Detailed endpoint - YouTube
+POST {{baseUrl}}/detailed
+Content-Type: application/json
+
+{
+  "fqdn": "youtube.com"
+}
+
+### Detailed endpoint - Reddit
+POST {{baseUrl}}/detailed
+Content-Type: application/json
+
+{
+  "fqdn": "reddit.com"
+}
+
+### Detailed endpoint - Twitter
+POST {{baseUrl}}/detailed
+Content-Type: application/json
+
+{
+  "fqdn": "twitter.com"
+}
+
+# ============================================
+# POST /full - Complete Raw Output
+# Returns: All 35 fields from NetStar with category groups
+# ============================================
+
+### Full endpoint - Facebook
+POST {{baseUrl}}/full
+Content-Type: application/json
+
+{
+  "fqdn": "facebook.com"
+}
+
+### Full endpoint - Google
+POST {{baseUrl}}/full
+Content-Type: application/json
+
+{
+  "fqdn": "google.com"
+}
+
+### Full endpoint - TikTok
+POST {{baseUrl}}/full
+Content-Type: application/json
+
+{
+  "fqdn": "tiktok.com"
+}
+
+### Full endpoint - YouTube
+POST {{baseUrl}}/full
+Content-Type: application/json
+
+{
+  "fqdn": "youtube.com"
+}
+
+### Full endpoint - Reddit
+POST {{baseUrl}}/full
+Content-Type: application/json
+
+{
+  "fqdn": "reddit.com"
+}
+
+### Full endpoint - Twitter
+POST {{baseUrl}}/full
+Content-Type: application/json
+
+{
+  "fqdn": "twitter.com"
+}
+
+# ============================================
+# Edge cases and special domains
+# ============================================
+
+### Basic endpoint - Unknown domain
+POST {{baseUrl}}/
+Content-Type: application/json
+
+{
+  "fqdn": "unknowndomainexample12345.com"
+}
+
+### Detailed endpoint - Localhost
+POST {{baseUrl}}/detailed
+Content-Type: application/json
+
+{
+  "fqdn": "localhost"
+}
+
+### Full endpoint - IP-like domain
+POST {{baseUrl}}/full
+Content-Type: application/json
+
+{
+  "fqdn": "99999.incompass.netstar-inc.com"
+}