texts: Random Forest Regression. How to represent really long list of categories for processing

texts:
I'm trying to build a model to solve a regression task.

Simplified, the data look like:

7.16, A 1, B 4, C 15, D 3
8.21, C 3, D 2, G  7, M 4, Y 9
1.85, D 3, N 1, L  1

The first column is a label, consequent columns are a class and respective number.

For example, the first row demonstrates that there is one of A, four of B, fifteen of C, and three of D - such a configuration gives us 7.16.

My Primary Question:

How do I represent my data so that I can fit a Random Forest Regression model?

My Thoughts:

Figure out the whole list of classes (there are 26 letters, but I have way more classes) and put zeros for all of the unknown classes into each record. Vector.sparse() might help here as it can reduce the amount of required memory.
Figure out the entire list of classes, correspond a number starting from 0 and convert each record into a vector (i.e. the first one would look): 0 1 1 4 2 15 3 3. Then each column is treated as a separate feature, but in this case I have to specify that some of my features are categorical and that's not possible with Spark since different records are of different length (In Spark I have to provide columns of features, which are categorical with number of classes. In the example that would be a map(0 -> 26, 2 -> 26, 4 -> 26) - like every second column is a class with 26 categories).
It's possible to build features like A:1, B:4, C:15, D:3 and treat all of them as categorical. Or move decision burden (categorical vs continuous) to Spark itself.


texts: Minimal (numerical) example for simple neural net

texts:
I‘m looking for a good minimal numerical example for teaching how a neural net is solved. Basically the idea is to see how the hidden layers (i.e. the weights) work using „pen and paper“.

Back in Uni I did similar things „by hand“ for OLS regression on a 3x3 matrix.

Does anyone know about a good source/example for neural nets?


texts: In a data analysis continuous data needs to be discretised, however in a computer it's impossible to have values $\in \mathbb{R}$, so what does this mean?


texts:
I am trying to import en_core_web_sm in my jupyter notebook.

import en_core_web_sm
nlp = en_core_web_sm.load()

error:

ModuleNotFoundError: No module named 'en_core_web_sm'


texts: Formatting categories of data with pandas in Python

texts:
I have around 9500 cities that I want to put in a pandas dataframe, then store in a file for later use. For example, below, these are 3 cities that I have data for. Some range in size based on year. Lovell_Wyoming only has 9 years of data points, corresponding to the years, as opposed to the 15 that Wheatland and Worland have.

My original idea was putting the quantitative data (Arson..Year) in a map, then putting the city name, as a key, into a larger map, with that quantitative data. Building the larger map that way. Then, converting the map to a dataframe, then to a csv. I am a bit inexperienced with pandas so I am not sure how to do this correctly, if this is even the best way to do it.

At the end of the day, I would like this data in a csv file that is easily accessible by loading it into a dataframe and calling on whatever value I need.

City 'Lovell_Wyoming'
Arson [0, 0, 0, 0, 0, 0, 0, 1, 0]
Assaults [6, 6, 3, 4, 3, 28, 3, 2, 2]
Auto_thefts [1, 1, 1, 0, 0, 1, 2, 0, 1]
Burglaries [6, 11, 5, 2, 0, 15, 11, 7, 7]
Murders [0, 0, 0, 0, 1, 0, 0, 0, 0]
Rapes [0, 0, 3, 0, 0, 1, 1, 0, 1]
Robberies [0, 0, 0, 0, 0, 0, 0, 1, 0]
Thefts [23, 49, 35, 39, 28, 37, 54, 35, 10]
Year [2002, 2003, 2005, 2006, 2007, 2008, 2009, 2010, 2014]

City 'Wheatland_Wyoming'
Arson [0, 0, 0, 0, 0, 1, 2, 0, 0, 0, 0, 0, 0, 0, 0]
Assaults [9, 2, 6, 5, 6, 6, 2, 4, 2, 4, 3, 11, 5, 4, 8]
Auto_thefts [4, 8, 3, 3, 4, 4, 5, 3, 4, 6, 4, 8, 12, 7, 3]
Burglaries [17, 17, 14, 9, 10, 17, 12, 26, 51, 12, 15, 21, 32, 31, 13]
Murders [1, 0, 0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0]
Rapes [0, 0, 0, 4, 2, 1, 2, 0, 2, 1, 1, 0, 2, 0, 0]
Robberies [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Thefts [109, 95, 146, 81, 108, 100, 82, 85, 106, 128, 48, 85, 66, 56, 47]
Year [2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016]

City 'Worland_Wyoming'
Arson [0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Assaults [12, 17, 19, 17, 11, 15, 16, 2, 9, 1, 4, 7, 2, 1, 3]
Auto_thefts [2, 1, 2, 1, 1, 8, 1, 1, 1, 0, 1, 0, 0, 0, 1]
Burglaries [6, 10, 10, 10, 9, 10, 10, 0, 6, 1, 0, 2, 0, 11, 18]
Murders [0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Rapes [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 3]
Robberies [0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Thefts [44, 41, 47, 29, 30, 25, 27, 27, 23, 30, 36, 45, 54, 46, 43]
Year [2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016]

I apologize in advance if the way I formatted this is a bit weird! Please let me know if you would like any more information.


texts: Keras Custom Layer calling fit method before compile

texts:
Below is the SS of the custom function I am trying to apply on every image of the batch and the custom Layer
def geo_features( input_img ):
    print( "INPUT IMAGE SHAPE:", input_img.shape, type( input_img ) )
    detector = dlib.get_frontal_face_detector()
    predictor = dlib.shape_predictor( config.SHAPE_PREDICTOR_DAT_PATH )
    cv2.imshow( 'test', input_img )
    cv2.waitKey( delay = 0 )
    img = cv2.resize(
        input_img,
       (
         config.TARGET_SIZE['CV2_LANDMARKS_RESIZE'][0],
         config.TARGET_SIZE['CV2_LANDMARKS_RESIZE'][1]
       )
    )
    row = [ ( -100, -100 ) ]
    faces = detector( img )
    .
    .
    .

class GeoFeatures( keras.layers.Layer ):
    def __init__( self ):
        super( GeoFeatures, self ).__init__()
        print( "BUILDING CUSTOM LAYERS")

    def call( self, inputs ):
        print( "INSIDE CALL METHOD" )
        return tf.map_fn( fn = tf.py_function( geo_features ), elems = inputs)

    def compute_output_shape(self, input_shape):
        return ( input_shape[0], config.NUM_GEOMETRIC_FEATURES )

The issue I am facing is, the GeoFeatures call method is being called before compilation as evident from the below pics:-

Below is part of the training code. (NOTE:- model_dispatcher.return_mode() is just returning the keras.Model and nothing else).
train_data = train_datagen.flow_from_dataframe(
            dataframe = train_df,
            directory = None,
            x_col = "Image",
            y_col = "Label",
            target_size=(config.TARGET_SIZE[config.MODELS[num_fold-1]][0], config.TARGET_SIZE[config.MODELS[num_fold-1]][1]),
            class_mode = "categorical",
            shuffle = True,
            color_mode = config.COLOR_MODE[config.MODELS[num_fold-1]],
            batch_size = config.BATCH_SIZE,
            seed = 42
        )

        val_data = val_datagen.flow_from_dataframe(
            dataframe = val_df,
            directory = None,
            x_col = "Image",
            y_col = "Label",
            target_size=(config.TARGET_SIZE[config.MODELS[num_fold-1]][0], config.TARGET_SIZE[config.MODELS[num_fold-1]][1]),
            class_mode = "categorical",
            color_mode = config.COLOR_MODE[config.MODELS[num_fold-1]],
            shuffle = True,
            batch_size = config.BATCH_SIZE,
            seed = 42
        )

        print( train_data.class_indices )
        print( "DISPATCHING MODEL" )
        model = model_dispatcher.return_model( num_fold )
        mc, reduce_lr = return_callbacks(
            num_fold,
            isBest
        )
        print( "NOT YET COMPILED")
        opt = return_opt( 'adam', learning_rate = 1e-3 )
        model.compile(
            optimizer = opt,
            loss = 'categorical_crossentropy',
            metrics = ['accuracy']
        )
        model.fit( .. )
        .
        .
        .
        .

As can be seen clearly, the compilation is not yet done still the call method of layer is being called resulting in an error.
For more clarity below is the model which is incorporating the custom layer. Full model code is not shown as it is irrelevant as the custom layer is used only once here. At the last, the customXceptionWithGeo returns Keras.Model( inputs, outputs )
def customXceptionWithGeo( TARGET_SIZE ):
    inputs = tf.keras.Input(TARGET_SIZE)

    y = GeoFeatures()( inputs )
    y = Dense( 16, activation = 'relu' )( y )
    y = Dense( 32, activation = 'relu' )( y )
    y = Dropout( 0.4 )( y )
    y = Dense( 64, activation = 'relu' )( y )
    y = Dense( 32, activation = 'relu' )( y )
    y = Dense( config.NUM_CLASSES )( y )
    .
    .
    x = ...
    .
    .
    x = layers.add( [ x, y ] )
    outputs = Activation('softmax', name='predictions')(x)
    return keras.Model( inputs, outputs )


texts:
I have a pre-trained model that I load after import gensim using model = KeyedVectors.load_word2vec_format('path', binary = True).
It has 46 000 words in 25 dimension vector space. I don't know anything about the model used to obtain such vectors.
I also have a 75-word watch list that all but one belong to model 's vocabulary.
I want to monitor tweets from ABC and XYZ for "matches" associated with my watch list.
How can I monitor scraped tweets for terms in my word list ? I do not know what to make of their vector representation in model, nor what to do with my word that's not in the vocabulary


texts: The validation loss < training loss and validation accuracy < training accuracy

texts:
I have a binary classification problem. I get the following results: a val_loss (far) lower than the train_loss, but the accuracy is also lower for the validation compared to the training set. How is that possible ?

Epoch 1/10
10708/10708 [=======] - loss: 0.6356 - acc: 0.8289 - val_loss: 0.4981 - val_acc: 0.7760
Epoch 2/10
10708/10708 [=======] - loss: 0.6243 - acc: 0.8248 - val_loss: 0.5075 - val_acc: 0.7609
Epoch 3/10
10708/10708 [=======] - loss: 0.6204 - acc: 0.8302 - val_loss: 0.5152 - val_acc: 0.7694
Epoch 4/10
10708/10708 [=======] - loss: 0.6215 - acc: 0.8307 - val_loss: 0.4981 - val_acc: 0.7824
Epoch 5/10
10708/10708 [=======] - loss: 0.6180 - acc: 0.8318 - val_loss: 0.4942 - val_acc: 0.7848

This is the Keras model I'm using:

model = Sequential()
model.add(LSTM(16, input_shape=(1, 1)))
model.add(Dense(2, activation='softmax'))

adam = Adam(lr=0.001)
model.compile(loss='categorical_crossentropy',
          optimizer=adam,
          metrics=['accuracy'])


texts: Implementing a Kernel Adaptive Filtering model explained in a paper

texts:
In this paper, Stock price prediction using kernel adaptive filtering
within a stock market interdependence approach, the authors propose a method for predicting stock prices by combining the predictions of Kernel Adaptive Filter (KAF) models trained on different stocks in different international stock markets. In their results, they compare this model against individual KAF models trained and inferred on individual stocks. The refer to these individual models as 'KAF-based methods' and refer to their model as 'Proposal'.

I am quite new to KAFs, so excuse any ignorance I may have in my questions.

The KAF-based models use various different variants of KAFs - KLMS, QKLMS, NICE, etc. However, when they are explaining their proposed method, they never seem to mention which KAF variant is used internally in their models.

On page 9 where they outline the pseudocode of their individual models using some KAF, line 7 says "compute the filter output" where the filter is defined simply as Kσ. Prior to this, on the bottom of page 5 and top of page 6 they give the initial definition of Kσ(·, ·) just as "a Mercer Kernel".

I am attempting to reproduce their results, and am trying to implement their method in Python, potentially using the KAF implementations from here. Which KAF should I be using as Kσ? KLMS? KRLS? Have I got the wrong understanding of what Kσ in the paper is?


texts: Structuring experiment/training data with months in mind

texts:
We're using a whole year's data to predict a certain target variable.The model works like data - OneHot encoding the categorical variables - MinMaxScaler - PCA (to choose a subset of 2000 components out of the 15k) - MLPRegressor. When we're doing a ShuffleSplit cross-validation and everything is hunky-dory (r^2 scores above 0.9 and low error rates), however in real life, they're not going to use the data in the same format (e.g. a whole year's data), but rather a subset that is due that month (e.g. in March they'll need the algorithm to predict rows that are due in March) and at this point, the algorithm fails.

The expectation is that we have some data (for example, 2018 December - 2019 December) that we're training on, and then we'll need to predict items due in January 2020 (that's not yet available). Again, if we do a random train-test-val split, the r^2 scores are great, but the team lead is not okay with this, as he said the real-life use case is that we train on 2019's data and then predict on a monthly basis, so he wants us to train on not all the data, but only 2018 December to 2019 November, and then validate on 2019 December data.

I see how this is methodically incorrect, as we're training on data that has different properties (whole year's data vs a certain month's due data), and this reflects on the validation scores. If I isolate a certain month to simulate the real life scenario (e.g. train on 11 months of data and test on one specific month), depending on the month we're getting r^scores between 0.3 and 0.7 depending on the month isolated (which is a far cry from the 0.9 if randomly sampled from the whole dataset).

What I can't figure out is, how can we structure this modelling so that the test data (items due in a specific month) has the same properties than the training set (the rest of the months, or maybe 12 months including 1 month from last year), while preserving information for the whole year? Or should we be doing 12 models for each month?


texts:
I'd like to find frequent patterns in data that has been created by an accelerometer of a Smart Watch.
The algorithm should return the parts of the data that occur after a pattern.
In the best case, the algorithm finds the movement automatically that shows how we open doors, grab our phone etc.

Unfortunately, until now I only was able to find algorithms that find frequent item sets in data.

Does anyone know a solution on how to find those patterns in continuous data?

Thank you very much! :)


texts: How to make predictions of multiple input samples at once in tf 2 with keras

texts:
I am quite confused on the output of model.predict

when after training I validate my model on around 6000 samples I use the following pseudo code:

model.fit(...)
predictions = model.predict(val_set)
len(predictions) # == len(val_set)

result: tensor array of shape=(len(tensor_array),14) (one prediction for each input sample)

in production I currently use the following code to ocr image numbers:

model = tf.keras.models.load_model('number_ocr_v2')
def predict(list_images):
    global model
    print("length:")
    print(len(list_images))
    #predictions = model.predict(list_images) # <- Same result
    predictions = model.predict_on_batch(list_images)
    print(len(predictions))
    print(predictions)

console Output:

  length:
  30
  {...tf loading text...}
  1
  tf.Tensor([[0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]], shape=(1, 14), dtype=float32)

(one prediction for x input samples)

Apparently the Model delivers one array with only the prediction of the first input tensor.

Where are the rest of the 30 Input Tensors? What is the difference between my training code and my production environment that explains why training code delivers predictions[:] and production code only delivers predictions[0]?

A Time consumption benchmark suggests that all 30 digits are beeing processed. Timing fits with 30 Predictions a ~70us per image

Best regards,

Julian


texts: LeNet-5 - combining feature maps in C3 layer

texts:
Famous LeNet-5 architecture looks like this:

The output of layer S2 has dimension: 10x10x6 - so basically an image with 6 convultions applied to it to derive features.

If each dimension was again submitted to 6 filters the resulting output would be of 10x10x36 however it is 10x10x16. Initially I stumble on it but finnaly I udnerstood that this is done be combining inputs from layer S2 and applying one kernel on it as it's explained in the article:

  Layer C3 is a convolutional layer with 16 feature maps Each unit in
  each feature map is connected to several 5x5 neighborhoods at
  identical locations in a subset of S2s feature maps

.

  The rationale behind the connection scheme in table I is the
  following The 1rst six C3 feature maps take inputs from every
  contiguous subsets of three feature maps in S2. The next six take input
  from every contiguous subset of four. The next three take input from
  some discontinuous subsets of four Finally the last one takes input
  from all S2 feature maps Layer C3 has 1,516 trainable parameters and
  151,600 connections

and the roadmap of it is provided in the table:

What I am still not uderstand is how exactly should I combine them?

In previous layer I've just applied 6 kernels on 1 dimension, resulting in 6 dimensions what was understandable. Here I am a bit lost to be honest :(

Please help.
Mateusz


texts: Let's say I already have a logistic regression model (or other) with N number of explanatory variables and is 70% accurate. Now if there are other variables available, how would I test if the new variables would improve my accuracy without building new model.


texts: What is the difference between majority vote, and greedy action in ensembling?

texts:
I read some stuff about majority vote and greedy action in ensembling, however, they kind of sound similar, but also different. What is the real difference between those two?

Thanks for your help!


texts: Recommendations for notable papers/analyses that make use of 'big' spatial data?

texts:
This is a reference request. I am seeking papers that make use of 'big' (subjective) spatial datasets in their analyses. While I am open to machine learning papers, I am also interested in papers that address more 'classical' spatial problems like aggregation, clustering, autocorrelation, and related topics. Bonus points if the paper is related to health!
Example papers:

Spatial big data analytics of influenza epidemic in Vellore, India
Spatial and temporal epidemiological analysis in the Big Data era (Review Paper)

If the question is too broad and/or inappropriate, please close.


texts: What is a good way to transform Cyclic Ordinal attributes?

texts:
I am having 'hour' field as my attribute, but it takes a cyclic values. How could I transform the feature to preserve the information like '23' and '0' hour are close not far.

One way I could think is to do transformation: min(h, 23-h)

Input: [0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23]

Output: [0 1 2 3 4 5 6 7 8 9 10 11 11 10 9 8 7 6 5 4 3 2 1]

Is there any standard to handle such attributes?

Update: I will be using superviseed learning, to train random forest classifier!


texts: "help" decision tree by tying 2 features together

texts:
Assuming I have in my dataset 2 (or more) features that are for sure linked (for example: feature B indicates the amount of relevance of feature A), is there a way I could design a decision tree that will tie both features together (the obvious split would be if feature B is big enough split by A and if not ignore A)?
In other words: I'm looking for a way to "help" my decision tree to not test feature A before testing feature B. Plus, not to look on the improvement of each feature individually, but on the improvement of "B and afterwords A".
Any help will be appreciated!


texts: How to compute accuracy for CNN when outputs are one-hot encoded

texts:
I want to compute accuracy of my model on test data. I have three classes so actual output might look like this :

[0 1 0]

and the predicted output might look like this:

[0.1 0.8 0.1]

How do I compute accuracy with this?
Shall I compute cross entropy and subtract 1 from that?


texts: Compute parameters of a PDF (probability density function) for which no closed form expression is available

texts:
I would like to compute parameters such as mean, variance, quantiles, etc. for a PDF which is only given as a piece of code. That is, it can only be evaluated numerically at given points; no closed-form expression.

For example, after using scikit-learn for kernel density estimation, I would like to compute parameters of the resulting PDF.

I feel that popular libraries such as numpy/scipy, scikit-learn, or pandas probably provide such functionality. However, I couldn't find it.

Can anybody please point me to it?


texts: I am trying to parse the files using Stanford nlp in Spark in mapper function. How to set the number of mappers in Apache Spark? Please help me.


texts: How can I show the relations between travel destinations?

texts:
I'm trying to do a project about email marketing. I'm working on a tourism company and I want to make a best destination suggestion for the clients. But I need to see the relations between destinations.

Example: How many people visited Dublin and then visited London?

My question: How can I best analyse this relation between the cities, given data about traveler itineraries?

I want to send email offers to clients who went to London and didn't go to Dublin (assuming a strong relation between London and Dublin).


texts:
I am pretty new for time series analysis and I would like to share one of my research questions.

Here is the graph:

Apparently, both 1 and 2 are time series data. 1 represents monthly humidity and 2 represents a morbidity rate.

I would like to analyze if the morbidity rate (2) is associated with seasonal humidity (1), what method is the best in this scenario? Also, from the visualization, it seems like the morbidity peak is about 3 months after the peak of humidity.

Can anyone give me some suggestions of this question?

Thank you for any replies!


texts: How to transition between offline and online learning?

texts:
I am training an RL agent on a time series (with TensorFlow in python) in the following way: to predict the quantity of interest at time period $t$, I feed a window of $W$ observations at time periods $[t-W,t)$. Throughout the training the window advances step by step until I have a minibatch of $M$ observations and rewards to train on. Repeat until you run out of historical data, that's one Epoch. I train on a few thousand epochs with small learning rate (the loss is very unstable).

Eventually, I want to start pulling live data from the environment to make predictions.
At this point, if I wanted to continue the training online, how should I deal with the epochs? There is no "running out of data" anymore.


texts: Method(s) suited for finding model parameters in supervised contexts

texts:
Assuming no particular knowledge, let us consider a system where a mapping $F:\mathbb{R}^n\times \mathbb{R}^k \to \mathbb{R}^{n} \times \{-1,1\}$ describes the transformations of $\mathbb{R}^n$ entities. Additionally, we are provided with a white box model for finding $\mathbf{y}\in \mathbb{R}^n$ such that $F(\mathbf{x},\mathbf{u}) = (\mathbf{y},c)$ for any input $(\mathbf{x},\mathbf{u})$.

Given $M$ training samples, $(\mathbf{x}_m,\mathbf{u}_m,c_m)$ with $\mathbf{x}_m \in \mathbb{R}^n$, $\mathbf{u}_m \in \mathbb{R}^k$, $c_m \in \{\pm 1\}$ such that $F(\mathbf{x}_m,\mathbf{u}_m) = (\mathbf{y}_m,c_m)$ for $m \in \overline{1\ldots M}$, what methods are there for performing the following tasks:

find $c^\ast$ such that $F(\mathbf{x}^\ast,\mathbf{u}^\ast) = (\mathbf{y}^\ast,c^\ast)$ for known $\mathbf{x}^\ast,\mathbf{u}^\ast$
find $\mathbf{u}^\ast$ such that $F(\mathbf{x}^\ast, \mathbf{u}^\ast) = (\mathbf{y}^\ast, 1)$

For the first task, I suppose that SVMs are one possible option, while for the second one regression could be tried.
Are there other classes of solutions that seem more appropriate for this type of problem?


texts: On simple 1D dataset, LogisticRegressionCV selects terrible hyperparameters, resulting scores are nonsensical

texts:
I am trying to use LogisticRegressionCV to fit a logistic regression model to a simple 1D dataset. Very oddly, when given a choice, it seems to select a tiny C value, which forces my model to select a tiny theta resulting in a useless model.
I tried looking at the scores_ provided by the model, but they make no sense. For example, when I tell it to do 3-fold cross validation with 5 choices of C values, it gives me:
{1: array([[0.47058824, 0.47058824, 0.47058824, 0.47058824, 0.47058824],
        [1.        , 1.        , 1.        , 1.        , 1.        ],
        [0.63636364, 0.63636364, 0.63636364, 0.63636364, 0.63636364]])}

The dataset is not linearly separable, and yet it claims to be getting 100% accuracy regardless of which C values I've given it to try.
Example code below:
import numpy as np
import pandas as pd
from sklearn.linear_model import LogisticRegressionCV
from sklearn.linear_model import LogisticRegression

def gen_y(x):
    p1 = np.clip(x + 0.5, 0, 1)
    v = np.random.uniform(0, 1)
    if v < p1:
        return 1
    return 0

np.random.seed(6)
x_data = np.sort(np.random.normal(0, 0.3, 100))
y_data = np.array([gen_y(x) for x in x_data])

regularized_logistic_regression_model = LogisticRegressionCV(Cs = np.array([10**-8, 10**-4, 1, 10**4, 10**8]), fit_intercept = False, cv = 3)

regularized_logistic_regression_model.fit(x_data.reshape(-1, 1), y_data)

print(regularized_logistic_regression_model.C_) # yields 10^-8
print(regularized_logistic_regression_model.coef_) # yields incredibly tiny value
print(regularized_logistic_regression_model.scores_) # yields nonsensical scores


texts: HMM - Matlab for data set to detect anomaly

texts: I have a dataset of oil temperatures. The time series consist of 100 hours of measurement at every second. There is an anomaly in the data that I would like to detect using Hidden Markov Models (HMM). In Matlab I used hmmgenerate, however I'm not able to link my data to the assumed probabilities. Instead random states and sequences are generated. Could anyone help me?


texts: LightGBM model improvement when the focus is on probability prediction

texts:
I am building a binary classifier using LightGBM. The goal is not to predict the outcome as such, but rather to predict the probability of the target even. To be more specific, it's more about ranking different objects based on the probability of the target event for them.
The dataset is imbalanced in that the distribution of classes is roughly 1 to 10. Not that the data is severely imbalanced, but this is definitely something that has its impact on the model's performance.
Given that probabilities are key for this task, I assumed that targeting the AUC score is more beneficial here especially given that it's somewhat immune to uneven class distributions.
I have a feeling that I didn't do a great job in feature engineering (I realize the importance of this part here), but let's assume for a moment that this is the dataset that I need to work with and all the feature engineering tricks have already been implemented.
Honestly speaking, I take it for granted that boosting-based models do not require much data wrangling. For instance, label encoding is enough and computationally expensive one-hot encoding can even be outperformed, etc.
With all that said, the results I get are far from perfect. Having an AUC score of 0.82 makes me think that in terms of probability prediction, the model is not awful, but the other metrics, as you can see, are satisfactory at best.
F1-score: 0.508
ROC AUC Score: 0.817
Cohen Kappa Score: 0.356

Analyzing the precision/recall curve and trying to find the threshold that sets their ratio to $\approx1$ yields a more balanced situation, but for this task, it's not yet clear which type of error should be minimized or whether, say, f1-score maximization is the target.
Anyway, all the conventional metrics are dependent upon the chosen threshold so it's not clear whether I can just save time on threshold tuning.
My questions:

Would it be correct to state that having a reasonably high AUC for such tasks can be prioritized as opposed to just looking at precision, recall and other metrics that are functions of thresholds?

I use a combination of Optuna and 5-fold cross-validation to select the best hyperparameters. The results, however, do not improve significantly. I cannot even get a very high AUC score on the train dataset regardless of the number of estimators used for LGBMClassifier.
Does it mean that this is some kind of plateau for this task, dataset, and features?
What are some common methods (in addition to better feature engineering and getting more data) to improve gradient boosting methods' results?

             precision    recall  f1-score   support

       False       0.92      0.76      0.83     10902
        True       0.40      0.70      0.51      2482

    accuracy                           0.75     13384
   macro avg       0.66      0.73      0.67     13384
weighted avg       0.82      0.75      0.77     13384

Results for threshold=0.66:
              precision    recall  f1-score   support

       False       0.89      0.89      0.89     10902
        True       0.52      0.51      0.51      2482

    accuracy                           0.82     13384
   macro avg       0.70      0.70      0.70     13384
weighted avg       0.82      0.82      0.82     13384

F1-score: 0.515
ROC AUC Score: 0.817
Cohen Kappa Score: 0.405
```


texts: Optimum values of all predictors in logistic regression

texts:
I am trying to figure out in logistic regression, with the help of coefficients of $x$ variables, we could figure out that on unit change in any $x$ variable what will be the change in probability of y variable.

But I want to understand what must be the values of different $x$ variables to be set so that we are most likely to get only desired results.

For example if we are trying to predict defaulters in loan dataset, now by running logistic regression and its coefficients, we would be able to get the idea that by unit change in $x_1$(or $x_2$ or $x_3$) what will be the change in probability of $y$ to be $1$.

Now if I want to understand that what must be the values of $x_1,x_2,x_3, \ldots$ etc to be set so that the chances of $y$ to be $1$ will be maximum?


texts: Decision trees and Curse of Dimensionality

texts: Since decision tree algorithm splits the training dataset one feature at a time, how the heck is possibly that it suffers from curse of dimensionality ?


texts:
I've studied this exercise in class

But I cannot figure out why, when I push down the Group By clause, I can remove RLCode attribute from GroupBy. Does this action change the meaning of query tree?


texts: What feature selection technique would you recommend for selecting many features (in the thousands)?

texts: I have a data set that has a large number of features (~8k features) and I want to limit the number of features so my model does not overfit but performs relatively well. I have mix of categorical and numeric features. I am trying to use these features for classification. Does anyone have any recommendations for methods that I can use that are not too computationally expensive? If you can steer me towards some literature or have an examples available that would be much appreciated.


texts: How to help neuronal network with an other model

texts:
I am working on an image classification problem, the input data normally is images to classify, but I thought latitude and longitude would play something on these satellite images. I sorted by latitude, but I could sort by x and y coordinates and applied an LSTM model, that proved coordinates has an influence on target.

How to add this as a neural network (LSTM) to the more important image classification neural network. Any hint or suggestion would be much appreciated.


texts: Lime Explainer: ValueError: training data did not have the following fields

texts:
I'm attempting to gather ID level drivers from my XGBoost classification model using LIME and I'm running into some odd errors. I'm using this link as a reference.

Here is the overall code that I'm using:

explainer = lime.lime_tabular.LimeTabularExplainer(Xs_train.values, class_names = [1.0, 0.0], kernel_width = 3)

predict_fn_xgb = lambda x: trained_model.predict_proba(x).astype(float)
data_point = Xs_val.values[5]

exp = explainer.explain_instance(data_point, predict_fn_xgb, num_features = 10)
exp.show_in_notebook(show_all = False)

Key:

trained_model: trained xgboost classification model
class names: This is a binary classification model
Xs_train: This is a (73548, 84) dimension training set. This was used to build the training_model
Xs_val: This is a (4910, 84) dimension training set. The columns are the same with the training and validation set.
data_point: one specific validation point

Now, when I run this code, I get the following error:

ValueError: expected res_time, email_views...training data did not have the following fields: f6, f49, f34, f21,...

I don't know where the f# column names are coming from. Seems really bizarre and I believe I'm following the example correctly.

Any help would be much appreciated. Let me know if any additional information is required.


texts:
I am working on multimedia enhancement tasks using deep generative neural networks. I only have access to one 11 GB GPU, and I really want to speed up my research. I am currently using a deep residual GAN, all GPU memory is occupied with a batch size of 8.

I want to experiment with using different architectures but every single trial takes at least 3 days. I have an idea about comparing multiple models in parallel, by reducing the model sizes, batch size or the input size. Once I find the best performing model, I can scale it up. Is this a bad idea? What are some other methods?


texts: Loss of MSE always be 0 when keras for topic predict

texts:
my input is a 200 dims vector, which is generated by mean of the word2vector of all words of a article, my output is a 50 dims vector,which is generated by the LDA results of a article I want to use mse as the loss function,but the value of the loss always be 0 my code as follows:

model = Sequential()
model.add(Dense(cols*footsize, 400,init = "glorot_uniform"))
model.add(Activation('relu'))
model.add(Dropout(0.2))
model.add(Dense(400, 400,init = "glorot_uniform"))
model.add(Activation('relu'))
model.add(Dropout(0.2))
model.add(Dense(400, 50,init = "glorot_uniform"))
model.add(Activation('softmax'))
model.compile(loss='mse', optimizer='rmsprop')

And regardless of the number of epochs, the loss is always 0


texts: Relating two data sets at the row level where table one may correlate to zero or more rows in another

texts:
I'm just looking for some high level recommendations on libraries, design patterns, or algorithms here. I obviously don't expect you to build a model for me.

I would like to predict the monetary impact that an outage will have on the electric grid.

I have data of historical outages that I could train a model on. This basically consists of the equipment-ID that was on outage, with a start and end timestamp.

An outage may cause zero or more constraints. A constraint is where a line becomes loaded to the extent that power must be flowed an alternate way. I already have data on which constraint-IDs occurred at what start and end timestamp, as well as the economic impact of that particular occurrence of that constraint. Note that a constraint may exist without any outages, so there is a bit of a loose relationship. Also, outages may occur without any constraints occurring, or it may take a combination of outages to cause a particular constraint. Generally speaking though, there is a moderate to strong relationship between outages and constraints.

I presume that there should be an algorithm of sorts, that can deduce which outages generally correlate to what constraints after they have been seen to overlap throughout history.

To recap, an outage may cause a constraint. I would like to feed the model outage data and constraint data, so that if I have planned outage data, it can predict if zero or more constraints are X% likely to occur.

I prefer to use Python, but am open to R.

How would you go about modeling a problem like this? In simplified terms, how do you correlate a row in one table to zero or more rows in another where there is no direct mapping? Most solutions I have seen correlate at the column level as opposed to the row.

Edit:

After some thinking about this, I've come up with what I think is a solution. Make each row with a timestamp for that particular hour of the day. One hot encode all outages for that hour. Also, create a column for each potential constraint, with the monetary impact as the value (Scaled to a 0-1 scale).

The thing that seems crazy to me, is that this data set is going to end up being thousands of columns wide if I do it this way. Are super-wide data sets like this common in ML? - or, is there a better approach?


texts: How is the hidden state of a GRU initialized

texts: This is a GRU. Now, what will be the value of $h_t$, at $t$=$0$. That is, what will be the value of the hidden state at just the starting?


texts:
i tried various techniques such as oversampling, undersampling, ROSE and both(oversampling and undersampling) on a imbalanced dataset to balance a dataset. when i applied all these techniques on a small dataset then these techniques perfectly work

> library(ROSE)

> table(df4$Price)

     0      1
100020   1249
> data.rose <- ROSE(Price~., data=df4, seed=3, N = 200040)$data
>  table(data.rose$Price)

     0      1
 99960 100080
# Oversampling
> data.balanced.over <- ovun.sample(Price ~ ., data = df4, method = "over",N = 200040)$data
 table(data.balanced.over$Price)

     0      1
100020 100020
> data.balanced.both <- ovun.sample(Price ~ ., data = df4, method = "both",N = 200040)$data
> table(data.balanced.both$Price)

     0      1
 99903 100137
> data.balanced.under <- ovun.sample(Price ~ ., data = df4, method = "under",N = 2538)$data
> table(data.balanced.under$Price)

   0    1
1289 1249

but i when i applied all these techniques on a large dataset then i got error

> table(mydata$Price)

       0        1
33003944  1150753
> data.rose <- ROSE(Price~., data=mydata, seed=3)$data
Error in omnibus.balancing(formula, data, subset, na.action, N, p, method = "rose",  :
  Too few observations.
> data.balanced.over <- ovun.sample(Price ~ ., data = mydata, method = "over",N = 66007888)$data
Error in (function (formula, data, method, subset, na.action, N, p = 0.5,  :
  Too few observations.
>  data.balanced.under <- ovun.sample(Price ~ ., data = mydata, method = "under",N = 2301506)$data
Error in (function (formula, data, method, subset, na.action, N, p = 0.5,  :
  Too few observations.

how to solve these errors?


texts: What makes a high quality image dataset?

texts: I am constructing a custom dataset for image classification. It is a simple task in the sense that only one class needs to be detected. However it is crucial that the dataset is high quality, and I am also researching ways to accomplish this. Some are clear such as images of the object up close and far away, but there might be scenarios that I haven't even thought of. In respect to the domain knowledge of object detection, what important situations must my dataset capture?


texts: Should inputs be shuffled for training a model with expected time dependency?

texts:
Consider a prediction model with numerical inputs and outputs.

Suppose data is inserted tick by tick, i.e., when new data is available it is inserted asynchronously. Current output depends on current inputs and past inputs and outputs.

Normally for training, inputs/outputs pairs are shuffled since no time dependency exists.

With such time dependency, should inputs still be shuffled?


texts: How many time a recommender system can recommand the same item to an user?

texts:
I'm working on an hybrid music recommender system project, my goal is to create recommendation playlists in accordance with users tastes.
I already implemented the first part which use a collaborative filtering algorithm, and I am now working on the content-based filtering part.

In order to make my recommender system accurate I read dozens of research papers talking about evaluating recommender system. There are many variables taken in account in those evaluations (such as coverage etc..)

Some of those papers talk about user confidence in recommender systems, defining it as one of the most important parameters to take in account. I read that for 10 items recommended if the user like something like 3 or 4 of them, it's enough to gain his confidence.

There is a point that i could'nt find in all those papers:
How many times can we recommend the same item to an user ?

To explain my question, during the playlists generation process there is a risk that the same music appear in two different playlists and I'm wondering what will be the impact on the users confidence in my recommender system in this case.


texts: Difference between explicit and implicit density with and without the relation to neural network

texts:
I have a lack of understanding about this issue. Could anybody explain it or give an advice to a good literature regarding it? I don't understand what is a explicit density model and how it differs from an implicit density one. Could anyone exemplify the difference between these two kinds of models?

Here is the picture source.


texts: Proportional Hazards Assumption - Schoenfeld Residuals X-Axis

texts:
I am testing the PH assumption for my COX PH model. Can someone describe what the two x-axis's are in the Schoenfield Residuals plots? One is rank-transformed time and the other is km-transformed time. How do I interpret these transformations (I'm not sure what these are practically)?
My response term here can range from counts of 1,000 to 30,000,000. Is this suggesting that within 150 (time counts), the coefficient increases to zero from ~ -1.0?


texts: Hyperparameter Tuning Time Series in Production

texts:
I have a time series data that handled using GDBT to predict the next value. I always use previous 30 days data to train daily, but overtime the data to predict and train is increased because the number of combination things increased.

My question is, how often do we need to hypertune our model? and what eval number combination considered to be enough for hyperparameter tuning? 50? 100? or just 10 is enough?

right now I do it  daily, but it getting costly and costly, previously it just aroung 10 minutes, but now about more than hour. The system need to do other things hourly so this hypertuning model will be a deadlock for the system.


texts: Options to find the most similar question in a dataset of question-answer pairs?

texts:
I am building a chatbot that will only handle FAQs, but these FAQs are very specific to an organisation, so I cannot use any existing off-the-shelf solutions, or connect to question-answering APIs.

I have a dataset which consists of questions, intents, and answers.

Let's say there are 100 intents, which basically group questions into general categories (e.g. fee_payment). Each intent has 50 different specific answers (e.g. 'Fees are paid through the online portal' or 'Fees are due on the 1st of every month'), and finally each answer has 10 sample questions (e.g. 'How do I pay my fees?' or 'What is the procedure for payment of fees?')

Here is an example of the structure of the dataset:

| intent      | answer                                  | question
-----------------------------------------------------------------------------------------------------
| fee_payment | Fees are paid through the online portal | How do I pay my fees?                      |
| fee_payment | Fees are paid through the online portal | What is the procedure for payment of fees? |
| fee_payment | Fees are paid through the online portal | .......................................... |
| fee_payment | Fees are paid through the online portal | <intent1_answer1_question10>               |
| fee_payment | Fees are due on the 1st of every month  | <intent1_answer2_question1>                |
| fee_payment | Fees are due on the 1st of every month  | .......................................... |
| fee_payment | Fees are due on the 1st of every month  | <intent1_answer2_question10>               |
| fee_payment | ....................................... | .......................................... |
| fee_payment | <intent1_answer50>                      | <intent1_answer50_question1>               |
| fee_payment | <intent1_answer50>                      | .......................................... |
| fee_payment | <intent1_answer50>                      | <intent1_answer50_question10>              |
| ........... | ....................................... | .......................................... |
| <intent100> | <intent100_answer1>                     | <intent100_answer1_question1>              |
| <intent100> | <intent100_answer1>                     | .......................................... |
| <intent100> | <intent100_answer1>                     | <intent100_answer1_question10>             |
| <intent100> | ....................................... | .......................................... |
| <intent100> | <intent100_answer50>                    | <intent100_answer50_question1>             |
| <intent100> | <intent100_answer50>                    | .......................................... |
| <intent100> | <intent100_answer50>                    | <intent100_answer50_question10>            |

I have started building the chatbot using Rasa and have the NLU part trained to accurately classify any arbitrarily worded question as one of the 100 intents. I now need to explore what is the best method to take all the questions in the dataset under this intent, and find the most similar/relevant one to the user's question.

What are the best options to acheive this? Would something as powerful as BERT be effective here? What are some other simpler lookup/search based options, or supervised/unsupervised ML/DL options, or ...?

Note: I am doing this in English initially, but eventually it will need to be language-agnostic (Arabic will be implemented next).

EDIT: The dataset also has a column of keywords for each question.


texts: How do we define a linearly separable problem?

texts:
When we talk about Perceptrons, we say that they are limited for approximating functions that are linearly separable, while Neural Networks that use non-linear transformations are not.

I am having trouble understanding this idea of linear separability. Specifically, does it apply only for binary classification or it generalizes for N classes? If so, how does a linear decision boundary for lets say 4 classes even look like?


texts:
Im studying the perceptron algorithm.
I know that we can use the weights w as the coefficients of the hyperplane that will separate the vectors that need a classifications

In every web page i read up for a detailed explanation, it's generally said that the perceptron algorithm will find the optimal weights w. But optimal in which sense ?


texts: How to make a region of interest proposal from convolutional feature maps?

texts:
Problem
Keras does not have any direct implementation of region of interest pooling. I am aware of how to perform maxpooling, but I don't know how to get bounding boxes from feature maps passed from convolutional layer.
Is there any way to directly implement a region proposal algorithm?

Example
Let's say there is an architecture like this:

So we have a multi-input neural network architecture that eventually leads to the ROI MaxPool layer. We have three inputs, screenshot, textmaps and candidates, let's take candidates out. Then we would have such code in Keras:
from keras.models import Model
from keras.layers import Input, Dense, Conv2D, ZeroPadding2D, MaxPooling2D, BatchNormalization, concatenate
from keras.activations import relu
from keras.initializers import RandomUniform, Constant, TruncatedNormal

#  Network 1, Layer 1
screenshot = Input(shape=(1280, 1280, 0),
                   dtype='float32',
                   name='screenshot')
# padded1 = ZeroPadding2D(padding=5, data_format=None)(screenshot)
conv1 = Conv2D(filters=96,
               kernel_size=11,
               strides=(4, 4),
               activation=relu,
               padding='same')(screenshot)
# conv1 = Conv2D(filters=96, kernel_size=11, strides=(4, 4), activation=relu, padding='same')(padded1)
pooling1 = MaxPooling2D(pool_size=(3, 3),
                        strides=(2, 2),
                        padding='same')(conv1)
normalized1 = BatchNormalization()(pooling1)  # https://stats.stackexchange.com/questions/145768/importance-of-local-response-normalization-in-cnn

# Network 1, Layer 2

# padded2 = ZeroPadding2D(padding=2, data_format=None)(normalized1)
conv2 = Conv2D(filters=256,
               kernel_size=5,
               activation=relu,
               padding='same')(normalized1)
# conv2 = Conv2D(filters=256, kernel_size=5, activation=relu, padding='same')(padded2)
normalized2 = BatchNormalization()(conv2)
# padded3 = ZeroPadding2D(padding=1, data_format=None)(normalized2)
conv3 = Conv2D(filters=384,
               kernel_size=3,
               activation=relu,
               padding='same',
               kernel_initializer=TruncatedNormal(stddev=0.01),
               bias_initializer=Constant(value=0.1))(normalized2)
# conv3 = Conv2D(filters=384, kernel_size=3, activation=relu, padding='same',
#               kernel_initializer=RandomUniform(stddev=0.1),
#               bias_initializer=Constant(value=0.1))(padded3)

# Network 2, Layer 1

textmaps = Input(shape=(160, 160, 128),
                 dtype='float32',
                 name='textmaps')
txt_conv1 = Conv2D(filters=48,
                   kernel_size=1,
                   activation=relu,
                   padding='same',
                   kernel_initializer=TruncatedNormal(stddev=0.01),
                   bias_initializer=Constant(value=0.1))(textmaps)

# (Network 1 + Network 2), Layer 1

merged = concatenate([conv3, txt_conv1], axis=-1)
merged_padding = ZeroPadding2D(padding=2, data_format=None)(merged)
merged_conv = Conv2D(filters=96,
                     kernel_size=5,
                     activation=relu, padding='same',
                     kernel_initializer=TruncatedNormal(stddev=0.01),
                     bias_initializer=Constant(value=0.1))(merged_padding)

If you look at the end of the code (and architecture itself), we pass concatenated activations from two different Conv+ReLu layers and then pass it to ROI MaxPool layer.

Thank you!


texts: Is smoothing in NLP ngrams done on test data or train data?

texts:
Is smoothing in NLP ngram done on test data or train data?
Since smoothing is to avoid the language model predicting 0 probability of unseen corpus (test). So I wonder is smoothing done on test data only? Or on train data only? Or both? I don't seem to find an answer to this yet.


texts: Getting feature vectors from CatBoost pool

texts:
I have a dataset with some numerical and categorical features and I am trying to apply CatBoost for categorical encoding and classification.

Since my dataset is highly imbalanced, with a large number of data samples with label 0 compared to those with label 1, I'm also trying to use SMOTE to synthesize label 1 data samples before CatBoost classification.

My code -

# train_categorical_features is a list of columns that have categorical values
train_pool = Pool(data = X,
                  label = y,
                  cat_features = train_categorical_cols)

X_enc = train_pool.get_features()
print(X_enc)
y_enc = train_pool.get_label()
print(y_enc)

smote = SMOTE()
X_res, y_res = smote.fit_resample(X_enc, y_enc)
print('Resampled dataset samples per class {}'.format(Counter(y_res)))

predictions = []
for i in range(10):
    clf = CatBoostClassifier(learning_rate=0.08,
                         depth = 10,
                         loss_function='Logloss',
                         l2_leaf_reg = 4,
                         iterations=1000,
                         task_type="GPU",
                         random_seed=i,
                         logging_level='Silent')
    clf.fit(train_pool, plot=True,silent=True)
    predictions.append(clf.predict_proba(test[inputcols])[:,1])
    print(clf.get_best_score())

I get an error on X_enc = train_pool.get_features() that says -

  CatBoostError: Pool has non-numeric features, get_features supports
  only numeric features

My questions are -

Is my approach towards applying SMOTE with CatBoost correct?
I've diligently followed the catboost documentation, and I am not able to understand or fix the error I've mentioned above. Would appreciate any help.


texts:
What is the difference between using numpy array images and using images files in deep learning?

Which way is better?


texts: Can someone please explain Lovasz softmax loss? as its a bit difficult to understand why it works well from the original paper

texts: Lovasz Softmax is used a lot these days for segmentation problem and the original paper is really bad at explaining why it works.


texts: How to quantify ‘compute cost’ of training of xgboost model?

texts:
I want to quantify compute cost of hyper-parameter search for xgboost model. One way can be to measure training time with one particular hyper parameter configuration chosen for training and use it as proxy for compute cost. Can we quantify compute cost based on hyper parameters of this model depending upon value of hyper parameters chosen e.g., analytical expression based on max depth, num of estimators, min child weight, gamma etc or can you suggest some other way to quantify this compute cost?
I want to measure for each particular hyper parameters chosen for training on same set of data what will be the model performance and compute cost!


texts:
I could not find anywhere how I can do this seemingly simple task (Retrieving column names in R was not helpful).
I have an R table defined as sports_by_sex <- table(ais$sex, ais$sport). This is using DAAG's ais dataset. If I want to get the name of the sport at sports_by_sex[, 2], how would I do that? When I print(sports_by_sex), this is my result:
    B_Ball Field Gym Netball Row Swim T_400m T_Sprnt Tennis W_Polo
  f     13     7   4      23  22    9     11       4      7      0
  m     12    12   0       0  15   13     18      11      4     17

So for the example of sports_by_sex[, 2], for example, I would like to get Field back.


texts: Making sense of indices in 2D convolution operations in convolutional neural networks


texts: Techniques to manage high dynamic range in neural network regression?

texts:
I have a problem where I take a noisy sequence in (in one domain) and generate a clean sequence out, in another domain.  The issue I'm having is that when the input data is relatively clean, the output can be generated with very high numerically accuracy, approximately 1e-6.  At some points in some sequences though, the input data is too noisy to do very well and the best possible error may be on the order of 1.0 to 10.0.
My problem is that while training using typical loss functions like MSE, the network is understandably "obsessed" with the high errors, with the result being that at no time does it achieve the very high accuracy obtainable in most cases.
Are there special loss functions or other techniques that can encourage the network to  go for highly accurate results when possible without completely punting on the hard cases?  I've played around with loss functions that only score the best X% of a batch, but so far it hasn't really helped much, and I still would prefer it to do the best it can on the hard cases.


texts: How can I make seaborn do PairPlot such that it wraps around?

texts:
I'm trying to do some PairPlots using seaborn, so I can compare a bunch of features against the label I'm trying to model. There are a bunch of features though (~50). So I'm basically doing:

g = sns.pairplot(train_df,x_vars=["MoSold","GarageArea","ScreenPorch","OverallQual"],y_vars="SalePrice",size=4)

Where "SalePrice" is the label that I want to compare the others against. For now I don't care about comparing the features to each other.

The problem I'm having, though, is that when I plot this in my Jupyter notebook, it formats it as one long row.

I'm only doing it with 4 features here, but when I add as many as I want to, it basically makes the row thinner and longer, and impossible to read (because it's trying to make them all share the same y axis).

What I'd like is for it to automatically wrap like panda's hist does:

I can think of a few little hacks. I could do several individual PairPlot commands to manually do it, using a for loop or something, but that would be kind of a pain. Is there some more elegant way to do this? Something where I'd specify the number of columns it'll display with, and then it'll make a grid, where the y axis is always SalePrice?


texts: What are the inputs to the first decoder layer in a Transformer model during the training phase?

texts:
I am trying to wrap my head around how the Transformer architecture works. I think I have a decent top-level understanding of the encoder part, sort of how the Key, Query, and Value tensors work in the MultiHead attention layers. What I am struggling with is the decoder part, specifically the inputs to the very first decoder layer.
I understand that there are two things. The output of the final encoder layer, but before that an embedded (positional encoding + embedding) version of... well something.
In the original paper in Figure 1, they mention that the first decoder layer input is the Outputs (shifted right). I am a little confused on what they mean by "shifted right", but if I had to guess I would say the following is happening
Input: <Start> How are you <EOS>
Output: <Start> I am fine <EOS>
and so the input to the first decoder layer will be [<Start> I am fine].
What is the need for shifting the sequence? Why would we not just input the target sequence itself? I was thinking maybe it's because of the auto-regressive nature of the decoder part, but then the only difference between the sequences would be the <EOS> token if I am seeing this correctly.
As you can probably tell I am a little bit confused by how some of the parts work, so any help to get to a better understanding would be much appreciated.


texts: keras Sequential CNN for image data reshaping data issues

texts:
I am new at keras and CNN and am working on building at CNN  for sequential analysis of movement in a image. What I am having issues with is the reshaping the data and the labels that go into the fitting and testing the data for the model.
So the original size/shape of the numpy file is (18, 50,50,16) which is saved in a text file from another program.  I know the text file is ok because I can read it in and display it correctly with the debug_potion method. So that looks good. There are 18 images in the folder for the data and otherData variable. I dont really know what the 16 is but the image size are 50*50.

The issue is the reshaping of that data is the problem. Can anyone suggest how to reshape this data in a way that I can train it. I think I need to do onehot encoding but not quite sure how. Any help will be appreciated. Here is what I have so far.

def debug_potion(data):
    for i in range(0, 16):
        # get for each joint
        potion = Potion(data)
        plt.show(potion.display(joints=[i, i], channels=[0, 1, 2]))

# Build Model for sequential building
def create_model(input_shape):
    model = Sequential()
    model.add(Conv2D(32, kernel_size=(3, 3),
                 activation='relu',
                 input_shape=input_shape))
    model.add(Conv2D(64, (3, 3), activation='relu'))
    model.add(MaxPooling2D(pool_size=(2, 2)))
    model.add(Dropout(0.25))
    model.add(Flatten())
    model.add(Dense(128, activation='relu'))
    model.add(Dropout(0.5))
    model.add(Dense(num_classes, activation='softmax'))

    model.compile(loss=keras.losses.categorical_crossentropy,
              optimizer=keras.optimizers.Adadelta(),
              metrics=['accuracy'])
    return model

def read_in_data(file):
    with open(file, "rb") as f:
        return pickle.load(f)

batch_size = 1
num_classes = 2
epochs = 12

# input image dimensions
img_rows, img_cols = 50, 50
input_shape = (img_rows, img_cols, 16)

data = read_in_data("heatmap.txt")
otherData = read_in_data("othermove.txt")

data = np.array(data)
otherData = np.array(otherData)
print("Data shape", otherData.shape)

xtrain = []
ytrain = []
xtest  = []
ytest  = []

for x in range(0,len(data)):
    if x < 15:
        xtrain.append(data)
        ytrain.append(("FWAC", num_classes))
    else:
        xtest.append(data)
        ytest.append(("FWAC", num_classes))

for x in range(0,len(otherData)):
    if x < 15:
        xtrain.append(otherData)
        ytrain.append(("Other", num_classes))
    else:
        xtest.append(otherData)
        ytest.append(("Other", num_classes))

#Create x test and train arrays
xtrain = np.array(xtrain)
xtrain = xtrain.reshape(30,img_rows, img_cols,16)
ytrain = np.array(ytrain)

xtest = np.array(xtest)
xtest = xtest.reshape(108, img_rows, img_cols,16)
ytest = np.array(ytest)
print(xtrain.shape)
print(xtest.shape)
print(ytrain.shape)
print(ytest.shape)

# Build Model
model = create_model(input_shape)

model.fit(xtrain, ytrain,
          batch_size = batch_size,
          epochs = epochs,
          verbose=1,
          validation_data=(xtest, ytest))

score = model.evaluate(xtest,ytest, verbose=0)
print(("Test loss", score[0]))
print("Test Accuracy", score[1])


texts: How to handle differences between training and deploying of an RL agent

texts:
Hi I am training an RL agent for a control problem. The objective of the agent is to maintain temperature in a zone. It is an episodic task with episode length of 10 hrs and actions being taken every 15 mins. Ambient weather is one of the state variable during the training. For training process a profile of ambient temperature has been generated for each hour of the day and used for training. I have trained the agent using PPO algorithm and the agent training is converging. I wish to deploy this model for a real world case and  have 2 questions regarding it.

If I train an agent for taking actions for 15 mins during the training process, is it ok I make my agent take actions at every 5 min interval during deployment?
If I train an agent on a particular ambient temperature profile, will the agent by heart / remember the profile used for training and expect the same temperature profile during deployment for it to work well ?

Can someone help me with these two things.

Thanks


texts: i need a characters /letters dataset for matlab

texts: My final project is number plate recognition.i need a data set of A-Z characters and 0-9 letters. i donot find it on any website give me a data set or send me a link. i have to make a neural network in which i put that data set to train and test my data.


texts: What NN architecture to predict fantasy character names based on description?

texts:
I would like to build a neural network to predict a fantasy character name given a description.

Like 'Scar-faced long haired elf warrior' -> 'Glorfindel'

I have a dataset of about 12,000 fantasy names and description from various fantasy works. I  want to be able to map the description to names. Names are not vocabulary words and I want to NN to be able to generate new names for new description.

I wanted to use something like Elmo to embed the description and the name which would then easily teach the NN to map one to another, but the problem I faced is how do I go back from an embedding vector to characters representing a word.


texts: Estimating probability using boltzmann machine

texts:
I am reading about Boltzmann machines and according the formulas the joint probability of the states of all units is $$ P(X = x) = \frac{1}{Z} e^{-\frac{1}{2T} \sum_i\sum_j {x_i x_j w_{ij}}} $$
$$ Z = \sum_x e^{-\frac{1}{2T}\sum_i\sum_j x_i x_j w_{ij}} $$
In thse formulas it is assumed that $ w_{ii} = 0$. However it is infeasible to calculate the quantity $Z$ directly. Therefore people use Gibbs sampling to sample different states. I don't understand how exactly the sampling is used to estimate probabilities.


texts: Image classification tool for detection of small features

texts:
I have a dataset of images of damaged cars. Each image has an associated mask of overall damage and severity index of each type of damage. Unet successfully predicts the overall mask of damaged area of a car. However, the model yields poor results when trying to detect small features like scratches. This is understandable, because most of the "positive mask" of a scratch contains other types of damage. Hence, if the model correctly predicts masks of the scratched areas it will be severely punished for not labeling most other pixels of the overall damage area.
What is the right tool to classify images, if each image has an overall label, but doesnot have a reliable mask for the scratch?


texts:
I am new to SVM (one-class) and was practically investigating it. Got some weird result that I can not explain. Let me demonstrate by some small reproducible code and visualization:
from sklearn.svm import OneClassSVM
import numpy as np
import pandas as pd
import plotly.express as px

training_data = np.random.normal(2,.5,(10000,1))
###testing data###
Y1=np.random.normal(5,1,(1000,1))
Y2=np.random.normal(4,1,(1000,1))
Y3=np.random.normal(3,1,(1000,1))

Fitting model
clf = OneClassSVM(gamma='auto').fit(training_data)

getting score for each data point
pred_training_score=clf.score_samples(training_data)
pred_y1_score=clf.score_samples(Y1)
pred_y2_score=clf.score_samples(Y2)
pred_y3_score=clf.score_samples(Y3)

getting prediction###
pred_training=clf.predict(training_data)
pred_y1=clf.predict(Y1)
pred_y2=clf.predict(Y2)
pred_y3=clf.predict(Y3)

####visualize the prediction on training data
df=pd.DataFrame(np.array([list(range(len(training_data))),training_data.reshape(-1),pred_training,pred_training_score]).T)
df.columns=['counter','values','labels','score']
fig = px.scatter(df, x="counter", y="values", color="labels",hover_data=["score"])
fig.show()

####training data visualization output

####Visualize testing data1 (with mean 5)
df1=pd.DataFrame(np.array([list(range(len(Y1))),Y1.reshape(-1),pred_y1,pred_y1_score]).T)
df1.columns=['counter','values','labels','score']
fig = px.scatter(df1, x="counter", y="values", color="labels",hover_data=["score"])
fig.show()

####testing data (mean 5) visualization output

This is great. Because I trained with data with mean2 and test with mean 5 and almost all test data has been classified with anomaly (label: -1).
The visualization of training data says that a soft margin has been computed and labeled data points near 2 as normal and other are abnormal/anomaly.
Now the interesting thing happens if I increase the spread of the data-points by increasing the standard deviation. So lets increase that and train again:
training_data = np.random.normal(2,1,(10000,1))

Now visualize the labeling of the training data:

The data points which are very close to 2 has been classified as anomaly. Why is that?
This gets worse if I increase the SD again:
training_data = np.random.normal(2,1.5,(10000,1))
and the visualization:

Could you please explain this type of behavior?


texts:
I have a dataframe with the following column

city <- c("Sydney NSW", "Newcastle NSW", "Liverpool NSW", "Broken Hill NSW")

I want to maintain everything prior to NSW (space included). What Regex expression can be used in R for that?


texts: How to train neural network using maximum likelihood principle

texts:
I often stumble upon papers saying that the NN was trained using maximum likelihood principle. I think I generally understand what maximum likelihood is. But I have trouble understanding how to apply it to neural network training.

How are neural network trained using maximum likelihood principle?


texts: What should be the value of "Embedding_initializer" flag while fine tuning pretrained embeddings

texts:
Need help to resolve my understanding of the Embedding layer parameters from keras. As per my understanding, providing an explicit embedding matrix as weight parameter and setting trainable flag to True would enable one to finetune pre-trained word embeddings.
Also I understand, the embedding initializer flag, initializes the embeddings for you and it takes the default value "uniform".
So, my question here is, what is the value that should be provided to embedding_initializer flag when fine-tuning pretrained embeddings when you intend to pass a weight parameter and set training flag to True.


texts: How is that possible that a reward function depends both on the next state and an action from current state?

texts:
There's the concept of "expected value of the next reward", often denoted as $\mathcal{R}^a_{ss'}$, and defined as

$$
\mathcal{R}^a_{ss'} = \mathbb{E}\left(r_{t+1}  \mid s_t = s, a_t = a, s_{t+1} = s' \right)
$$

which is the value we expect for the reward at the next time step, that is at time step $t+1$, given that action $a$ from state $s$ brings us to state $s'$.

But the italicized definition does not seem to be "consistent" with the Bellman equation for the value function, which is

\begin{align}
V^\pi(s) &= \sum_{a \in \mathcal{A}(s)} \pi(s, a) \sum_{s' \in \mathcal{S^+}} \mathcal{P}^a_{ss'} \left( \mathcal{R}_{ss'}^a + \gamma V^\pi(s') \right)
\end{align}

Why do I say this?

$\mathcal{R}_{ss'}^a$ depends on $s$, $a$ and $s'$, but in the Bellman equation above, specifically in the inner sum $\sum_{s' \in \mathcal{S^+}} \mathcal{P}^a_{ss'} \left( \mathcal{R}_{ss'}^a + \gamma V^\pi(s') \right)$, we are essentially iterating over all possible next states, $s'$, of $s$. But, if that's the case, then what's the purpose of making $\mathcal{R}_{ss'}^a$ depend on the action $a$? That is, if we already have the next state $s'$, then the action is useless, given that we are already taking into consideration the possible next states, indeed. In other words, $a$ and $s'$ may not be compatible.

For example, suppose we are in a certain state $s$, and we take action $a$ from there, and we end up in a state, say, $x$, which must be different to all next states $s'$ (in the inner summation), apart at most from one.

I hope you can see my point and my doubts, and I hope you can clarify them.

Furthermore, given this doubt, I am unable to implement a reward function. Also, I have seen an implementation of the reward function, that is

def reward(state, next_state, action):
    if GRID[state] != GOAL and GRID[next_state] == GOAL:
        return 10.0
    else:
        return 0

which actually does not use the action, but I don't understand why. Why should we use next_state and not the next state of state according to action (or vice-versa)? How could (and why would) we use both next_state and action, if they may be "incompatible" in the sense I described above?


texts:
How can I change the table format from this:

To This?:

Is there any way to do this in Python using pandas or NumPy?


texts: Can identifiers be used to train a model?

texts:
I recently participated in some Machine Learning competition where we were asked to decide whether a rider should accept or not a course (~2k riders and ~140k courses). It came up that some of the winners used the identifier of the rider (an integer number unique for each rider) in their features which was discarded on default notebook and it greatly improved their score.
Is it legitimate? can identifier be used to train a model?


texts: With the Start Date and End Date given , finding date counts in Pandas


texts: What causes the error in a RNN to increase late in training?

texts:
I'm training a 2-layer, 1024 node, dropout of 0.5 RNN over natural text. Specifically, I'm using karpathy's char-rnn which I found to work quite well for most of my use cases. Sometimes however, late in the training, my errors drastically increase, and I can't figure out why this is happening.

The question is specifically asking why am I observing the drastic increase in error, and how this is handled in practice. What can be done to prevent this besides taking the last-known good model?


texts:
I am interested in the question, is there any research on the recognition of numbers from audio?
For example, there are call records and you need to get the numbers that speak on the phone.


texts: EarlyStopping in GridSearch - how to get the mean epoch after which training stopped?

texts:
is there a way to get the mean number of epochs when training stopped by EarlyStopping in GridSearch?

grid = GridSearch(estimator=model, param_grid=params, cv=10, scoring='accuracy', refit='acc')

results = grid.fit(X, y, epochs=200, valiation_data=(X_val, y_val), callbacks=[EarlyStopping(monitor='val_loss', patience=10, verbose=0, mode='min', restore_best_weights=True)])

Something like that:

for mean, stdev, param in zip(grid.cv_results_['mean_test_acc'],
                          grid.cv_results_['std_test_acc'],
                          grid.cv_results_['params'],
                          grid.cv_results_['mean_epochs' ?????

Many thanks for every hint


texts: In Affinity Propagation do the cluster centres minimize mean distance to all other points in the cluster?

texts: For Affinity Propagation, do the cluster centers minimize the mean distance to all other points in the cluster?


texts: Why do we add "αd" to N in Laplace Smoothing?

texts:
I just started to learn Naive Bayes algorithm. Then I learned to use Laplace smoothing to avoid getting probability of zero.
I understand the purpose of using it, but, in the expression of Laplace smoothing below, I do not really understand why we need to add "αd" to N in the denominator. As far as I researched, one of websites says that it is done so that the division will never be greater than 1. However, I still do not understand why we add "αd". Could someone explain why we have to add "αd" to N in the denominator ?


texts: Previous work Replication and Research ethics Ask Question

texts:
I am very much concerned about biding by research ethics in my work, especially issues to do with plagiarism. I come across a recent research paper in my field of study that applies state-of-the-art tools (deep learning architectures) in their work using a publically available dataset.

I am impressed by their work and feel I should apply the same methodology they used but using my dataset (private).

Would this be considered a plagiarised version of their work?


texts: Recommended minimum number of shared nodes between snapshots in temporal community detection

texts:
A little context
Social networks such as Facebook, Twitter or Reddit can be represented as dynamic temporal communities. Temporal networks consist of snapshots, representing the state of the network within a given time-frame (for example: nodes = Reddit users, edges = interactions between users, time-frame = 15 minutes). The easiest method to employ community detection on temporal networks is to apply any static community detection algorithm (such as Louvain) on each snapshot and extract the static communities - then use them to extract various information.
The problem
I sampled a dataset consisting of all Reddit comments on the politics subreddit 2 days before and 3 days after Biden's inauguration.
I chose a time-frame size of 15 minutes.
The problem is any two snapshots in this dataset have a really small number of shared nodes, let's say less than 10% of each's snapshot's total number of nodes. Thus, my snapshots form separate, stand-alone static communities rather than a temporal network.
So, is this normal? Is there any way to sample a set of comments in order to maximize the shared nodes number?
Is there a recommended minimum number of shared nodes between snapshots in temporal community detection?


texts: Build text complexity model based on complex examples

texts:
I try to build the user specific model which predicts whether arbitrary English text is complex for particular user or not. Having the complex and easy text samples allows to build such model but what if I have only complex samples. How can I build the model in such case?

I can detect whether the given text is different (find the "outlier") from those which user marked as difficult. But that information does not tell me in which way it's different. The text could be easier or more difficult.

Currently I see only one way - make an assumption about how the easy text could look like. But it's kind of unsafe since different people might have own unique areas which they do not understand in the text.


texts: Finding IDs that have been consistently failing to meet target

texts:
It's my first time here on stack exchange.
I have table with data columns including ID, Domain, Target and Dates (The table seen below). I am currently performing analysis on it and having trouble in finding which IDs have been failing consistently to meet the target.

Code

# importing pandas as pd
import pandas as pd

# Create sample dataframe
 raw_data = {'ID': ['A1', 'B1', 'C1', 'D1'],
'Domain': ['Finance', 'IT', 'IT', 'Finance'],
'Target': [1, 2, 3, >1],
"1/01":[0.9, 1.1, 2.1, 1],
"1/02":[0.4, 0.3, 0.5, 0.9],
"1/03":[1, 1, 0.5, 1.1],
"1/04":[0.7, 0.7, 0.1, 1],
"1/05":[0.7, 0.7, 0.1, 1],
"1/06":[0.9, 1.1, 2.1, 1]}

 df = pd.DataFrame(raw_data, columns = ['ID', 'Domain', 'Target', '1/01',
'1/02','1/03', '1/04','1/05', '1/06'])

The main question that i must answer is

Which ID has been failing consistently, How many times?
Which ID have been meet consistently, beating the target very comfortably?

Can someone please help me ? I'm really new to this pandas and data analysis


texts: How to adjust cofounders in Logistic regression?

texts:
I have a binary classification problem where I apply logistic regression.

I have a set of features that are found significant.

But I understand that Logistic regression doesn't consider feature interactions.

While I read online that a lack of feature interaction can be accounted by adjusting logistic regression for confounders.

Currently I did this and got the significant features.

model = sm.Logit(y_train, X_train)
result=model.fit()
result.summary()

But how do I adjust for confounders? Any pointers to tutorial for non-stat person like me would really be helpful.

Can someone let me know on how can I do this?


texts:
I recently read Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network by Wenzhe Shi et al. I cannot understand the difference between deconvolution (mentioned in Section 2.1) and the Efficient sub-pixel convolution layer (ESCL for short) (Section 2.2)

Section 2.2 defines the ESCL as performing convolution with a fractional stride. Isn't this what deconvolution is? Why is a distinction being made?

Thanks


texts:
I have a need to create a model that compares and groups distinct snippets of text based on keywords.  I can extract similar keywords with NLP methods and simply comparing sentence text.  I want these similar sentences to arise based on new text a user is entering and I want to give the user the ability to indicate whether or not the chosen sentences are actually similar to their sentence (would limit to 5 or so most similar).  The user's rankings would then feed along with the NLP key words into a machine learning model that would continue getting updated each time a new text entry is processed from a user.
I have two main questions:

I work in an environment which generally uses C# for most projects.  Is there a way I could accomplish the above with ML.NET?  I've seen a lot of basic supervised binary classification examples, but it seems to me that what I'm attempting to accomplish would be better suited as unsupervised where I am simply feeding in keywords, rankings, and text and trying to establish patterns.  If not ML.NET, is there another easy way to reference external models, such as those created in Python in a .NET environment?

Breaking down the text classification / comparison seems like it would be best suited to something like a bag of words and performing clustering based on those words, but also seems like it should be focused on only the key words to keep the model smaller/simpler.  How would I account for rankings as well in this case?  I'm not very familiar with text classification algorithms, so was hoping someone with more experience could point me towards the algorithms that might work best in this scenario?


texts: Are more classes more favorable than a single combined class?

texts:
Imagine the following scenario.

Train a classifier that classifies an object into one of these n+m classes:

class 1..n     -> triggers an action A depending on the predicted class p
class n+1..n+m -> triggers an action X independent on the prediction

Used is the CalibratedClassifierCV of a LinearSVC of scikit-learn. So, the prediction is based on the maximum predict_proba value.

Question

Because action X is independent on the prediction, I would very like to combine the m classes (n+1, n+2, ..., n+m) into a single class (to improve training speed, classification speed, memory consumption etc.)

Would that harm the classification result?


texts: Derivation of HiddenState wrt Output of LSTM

texts:
I'm busy trying to understand the math behind LSTM RNN's. In most of the math tutorials that I've found the derivations (Backpropagation) don't consider a dense layer before the output, instead they consider the final hidden state of the cell as the output of that cell. So hence the derivation starts from the hidden state as $EDelta = Y(x) - h(x)$ (where Edelta is gradient of Hiddenstate as "output" of Cell, Y(x) is target vector, h(x) is hiddenstate) and then use that $EDelta$ for the backpropagation for that specific cell. What I am trying to figure out is, if I am using a dense layer and a transformation of the Dense layer as a final output, would the gradient, specifically $EDelta$, look as follows:
Consider my final output as $Z(x)$ , my dense layer as $D(x)$, weights connecting $D(x)$ to $Z(x)$ as $W_dz$.
(I am unsure if there are supposed to be weights connecting my hiddenstate to the dense layer or if each unit/pair of units in my dense layer are simply the corresponding hiddenstate unit/units, I am considering the latter for now, please correct if I'm wrong).
Derivation:
$\nabla Z(x) = Y(x) - Z(x)$
$\nabla W_dz = \nabla Z(x)D(x)$
$\nabla D(x) = W_dz^TZ(x)$
$EDelta = \nabla D(x) \circ g^!(D_p)$ (where $g^!$ is activation derivative and $D_p$ is the preactivation value)
and then my derivation within the specific cell would continue as normal (ie. Outputgate($O$) gradient = $EDelta ^* tanh(C_t)$ where $C_t$ is Cellstate.
For reference, I got the backpropagation formula that I'm referring to from https://www.geeksforgeeks.org/lstm-derivation-of-back-propagation-through-time/
(Just for context incase I am completely wrong, I started coding about 4 months ago, which is also when I started learning how to read formulas, I almost failed maths at school (5 years ago) so I know I have far to go but I am adamant to learn from the roots up so this is why I'm trying to understand the math).


texts: List value in Pandas DataFrame column makes analysis harder

texts:
Should I move to database?
I have a list of courses data in JSON format that looks like this:
courses = [
  {
    course_id: "c_01",
    teachers: ["t_01", "t_02"]
  },
  {
    course_id: "c_02",
    teachers: ["t_02", "t_03"]
  }
]

And a list of teachers that look like this:
teachers = [
  {
    teacher_id: "t_01",
    teacher_fullname: "teacher_01"
  },
  {
    teacher_id: "t_02",
    teacher_fullname: "teacher_02"
  },
  {
    teacher_id: "t_03",
    teacher_fullname: "teacher_03"
  }
]

I also have other data like courseworks and submissions that I want to cross with these to create summary analytics like:

¿Which are the courses of a teacher?
¿Who is the teacher with most courses?
¿What is the average coursework quantity per course per teacher?
etc...

I loaded each list (courses, teachers, courseworks, submissions) into DataFrames, but I'm having a hard time selecting the courses of a teacher using vectorized methods.
Programming attempts

Tried using DataFrame.query but failed to use array methods inside the query
Tried using Series.isin but as it can't hash the list inside the Series teachers is useless.

Data manipulations attempts
Then I tried to flatten the teachers Series as follows:
courses = [
  {
    course_id: "c_01",
    teacher_id: "t_01"
  },

  {
    course_id: "c_01",
    teacher_id: "t_02"
  },
  ,
  {
    course_id: "c_02",
    teacher_id: "t_02"
  }

  {
    course_id: "c_02",
    teacher_id: "t_03"
  }
]

Which works because it allows merges and all kinds of queries, but because of combinatorial explosions I ended up with thousands extra rows I no certainty that the aggregated numbers are correct.
My last approach was to add one column per teacher in each course like this:
courses = [
  {
    course_id: "c_01",
    teacher_01: 1,
    teacher_02: 1,
    teacher_03: 0,
  },
  {
    course_id: "c_02",
    teacher_01: 0,
    teacher_02: 1,
    teacher_03: 1,
  }
]

It actually works perfectly and the aggregations become very easy to do. One concern is that I may end up with thousands of columns (I'm not sure if that's a problem) and the other is that each time a run an analysis on a batch of data I'll end up with different columns so it will be harder to make cross analysis between different datasets.
Anyway, my final thoughts are that I should store everything in a database and query the information that I need already "joined" to perform an easier and cleanier analysis process.
so...Should I move to database? Am I missing something?


texts: Coding a Content Addressable Memory on a GPU

texts: I´m trying to code a CAM or more simply a dictionary storing the pointer of the data accessible by a key. I try to do it with a GPU but all attempts have been inefficient compared on using System.Collections.Generic.Dictionary. Does anybody know how to implement this with CUDA to obtain a better performance than with a CPU?


texts: Issue with Custom object detection using tensorflow when Training on a single type of object

texts:
I am training a pre built tensorflow based model for custom object detection. I want to detect only 1 type of object. I have taken lot of images from different angles and in different light conditions. I am training on K80 Nvidia GPU. Everything is working and when I train I can see the loss function falling to 0.3. But the loss values drops very quickly to under 1 when I start training. I am using SSD mobile Net as the base configuration for the model. When I try to test the model, it just draws a big square on the input image, rather than detecting the desired object in the image. Basically, it fails to detect the object.

I tried to train the model with a different set of images of mac n chesse which had lot of variations. Then the model worked fine and detected images of mac n chesse in the input image. But when I have pictures of single object then the model fails to detect. Please help me understand what I am doing wrong here


texts: What is a sliding-window convolutional neural network?

texts:
In the abstract of "U-Net: Convolutional Networks for Biomedical
Image Segmentation", the authors mention a sliding-window convolutional neural network. I've found several other articles mentioning this but no clean definition of it. How is it different from regular convolutional networks/what does the term mean?


texts: Fine-tuning a CNN for recognizing two classes, but also being able to tell if none of them is present in an image

texts:
I need to fine-tune a CNN to classify two classes: dogs and cats, for example. However, I want the CNN to be able to tell if there are no dogs nor cats in a given image. Hence, I'm thinking of using a third class called background.

The goal is to fine-tune the network with lots of images: images of cats go to the cat class, images of dogs go to the dog class, and every other image goes to the background class.

This way, the fine-tuned network would be able to classify a dog as a dog, a cat as a cat, and (hopefully) everything else as background.

Is this the right way to do it? Would it work? I can't seem to find reliable information about this online.

My problem is a little bit more complicated, but knowing this would be a very good start.


texts:
I'm using k-means clustering to processes running on machines.

Dataset sample :

machine name, process
m1,java
m2,tomcat
m1,word
m3,excel

Build a matrix of associated counts :

   java,tomcat,word,excel
m1,1,0,1,0
m2,0,1,0,0
m3,0,0,0,1

I then run k-means against this dataset (have tried Euclidean and Manhattan distance functions)
The dataset is extremely sparse which I think is causing the generated clusters to not make much sense as many machines get grouped into the same cluster(as they are very similar)

How to achieve clusters where each cluster contains approx equal number of points ? Or perhaps this is not possible due to the sparseness of the data and instead I should try to cluster on a different attributes of dataset ?


texts: Finding value of theta in linear classification

texts:
I have an examing coming up, and I'm practicing with exams from previous years. However, the answers to the questions are not provided unfortunately.
I'm currently doing the question below, and the intuitive answer I get is 2. However, this seems too easy. Am I correct?


texts: How is WordPiece tokenization helpful to effectively deal with rare words problem in NLP?

texts: I have seen that NLP models such as BERT utilize WordPiece for tokenization. In WordPiece, we split the tokens like playing to play and ##ing. It is mentioned that it covers a wider spectrum of Out-Of-Vocabulary (OOV) words. Can someone please help me explain how WordPiece tokenization is actually done, and how it handles effectively helps to rare/OOV words?


texts: Seems like the Keras .fit and .evaluate methods give different training accuracy (yet the same validation accuracy). Same thing on loss.?

texts:
The code below is a translation of Nielsen's first mnist code to Keras.  Surprisingly, the last accuracy value of the .fit method and the accuracy value for the .evaluate method are different for the training data.  As I expected, for the validation data the accuracies are the same. The same behavior is seen with losses.  I would appreciate finding out what is going on.
import numpy as np
from tensorflow import keras
from tensorflow.keras import layers

num_classes = 10
input_shape = (28, 28, 1)

# the data, split between train and test sets
(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()
print(x_train.shape,y_train.shape,x_test.shape,y_test.shape)
# Scale images to the [0, 1] range
x_train = x_train.astype("float32") / 255
x_test = x_test.astype("float32") / 255
# Make sure images have shape (28, 28, 1)
x_train = np.expand_dims(x_train, -1)  #adds another dimension at the end for color
x_test = np.expand_dims(x_test, -1)

# convert class vectors to binary class matrices
y_train = keras.utils.to_categorical(y_train, num_classes)
y_test = keras.utils.to_categorical(y_test, num_classes)

initializer = keras.initializers.RandomNormal(mean=0., stddev=1.)

model = keras.Sequential(
    [
        keras.Input(shape=input_shape),
        layers.Flatten(),
        layers.Dense(30, activation="sigmoid", use_bias=True,
                     kernel_initializer=initializer,
                     bias_initializer=initializer),
        layers.Dense(num_classes, activation="sigmoid",
                     kernel_initializer=initializer,
                     bias_initializer=initializer),
    ]
)
batch_size = 10
epochs = 2
opt = keras.optimizers.SGD(learning_rate=3)
model.compile(loss="MeanSquaredError", optimizer=opt, metrics=["accuracy"])
ann=model.fit(x_train, y_train, batch_size=batch_size, epochs=epochs, validation_split=1/6)

# Evaluate the trained model
print("\n")
score = model.evaluate(x_train[:50000], y_train[:50000], verbose=0)
print("evaluate train accuracy:", score[1])
score = model.evaluate(x_train[-10000:], y_train[-10000:], verbose=0)
print("evaluate validation accuracy:", score[1])
print("history last train accuracy:",ann.history['accuracy'][-1])
print("history last validation accuracy:",ann.history['val_accuracy'][-1])


texts: Estimating data set size for grammar extraction

texts:
I have a dependency treebank including 100 sentences, which I divide into a training set and a test set. I extract some rules ((DS,PS) pairs) to convert the treebank to phrase structures. When I extract such rules from the training set, I can measure the percentage of rules (DS patterns) that cover the test set, suppose

(10, 24%), (20, 34%), (30,40%), (40,44%), (50, 55%),(60, 58%), (70, 61%)...

As you see as I increase the size of the training set, the coverage of extracted patterns increases! however its not linear!, I want to see how many data I need to reach 100% coverage?
I guess I can use a regression, but which regression? logarithmic?

Is this related to 'learning curve'? if yes how can I use regression for a learning curve?


texts: confused about parameter updates and forward/backward pass according to batches and epochs in CNN?

texts:
I am working on a CNN model, the code written in tensorflow, I did some googling about parameter updates such as weights ana biases when method is optimized and the loss is computed, two things made me confuse:

1- after output layer, the data goes to loss, the loss compute and then the model is begin optimization or in reverse of that?

2- Is parameters updated after each mini-batch fed to network (i.e. the forward and backward pass is done for every batch) or only updates when one epoch is completed? why some tutorials said that each epoch is ba forward/backward pass?

anyone can clarify it please? if with a reference its better for me.


texts: Regression Model for explained model(Details inside)

texts:
I am kind of a newbie on machine learning and I would like to ask some questions based on a problem I have .

Let's say I have x y z as variable and I have values of these variables as time progresses like :

t0  = x0 y0 z0
t1  = x1 y1 z1
tn  = xn yn zn

Now I want a model that when it's given 3 values of x , y , z I want a prediction of them like:

Input : x_test y_test z_test
Output : x_prediction y_prediction z_prediction

These values are float numbers. What is the best model for this kind of problem?
Thanks in advance for all the answers.

More details:
Ok so let me give some more details about the problems so as to be more specific.

I have run certain benchmarks and taken values of performance counters from the cores of a system per interval.

The performance counters are the x , y , z in the above example.They are dependent to each other.Simple example is x = IPC , y  =  Cache misses , z  = Energy at Core.

So I got this dataset of all these performance counters per interval .What I want to do is create a model that after learning from the training dataset , it will be given a certain state of the core ( the performance counters) and predict the performance counters that the core will have in the next interval.


texts: Given the files and their parsed content how can I reverse engineer the file format?

texts: I have access to a few thousand files binary files and I don't have the program that generated them. The files have been parsed and I also have access to the database where the parsed fields are stored. What techniques can be used to figure out the format of the files, given this information?


texts: How can we effectively measure the impact of our data decisions

texts:
Apologies if this is very broad question, what I would like to know is how effective is A/B testing (or other methods) of effectively measuring the effects of a design decision.

For instance we can analyse user interactions or click results, purchase/ browse decisions and then modify/tailor the results presented to the user.

We could then test the effectiveness of this design change by subjecting 10% of users to the alternative model randomly but then how objective is this?

How do we avoid influencing the user by the model change, for instance we could decided that search queries for 'David Beckham' are probably about football so search results become biased towards this but we could equally say that his lifestyle is just as relevant but this never makes it into the top 10 results that are returned.

I am curious how this is dealt with and how to measure this effectively.

My thoughts are that you could be in danger of pushing a model that you think is correct and the user obliges and this becomes a self-fulfilling prophecy.

I've read an article on this: http://techcrunch.com/2014/06/29/ethics-in-a-data-driven-world/ and also the book: http://shop.oreilly.com/product/0636920028529.do which discussed this so it piqued my interest.


texts: Deep CNN with variable number of classes and "vanishing" data

texts:
I am using a deep CNN to predict the class an image belongs to (N classes). However, the number of classes is not stationary. I.e. over the time the network will be used, some new classes may emerge while others may not be relevant anymore (I do not need to identify these in any way but I know the relevant classes at any time step).

Additionally, I am facing the constraint that the data I can use for this task "vanishes". This means that, at a point in time, I can only use the data of the previous M time steps (e.g. months) where M remains constant. This prohibits simply retraining the network on the entire dataset here and then while dropping some classes/adding new classes (i.e. changing the size of the output layer).

I was thinking of using the feature extracting part of the CNN to generate the feature vector and then use binary classifiers (e.g. logistic regressions) on that. This would enable me to simply discard a binary classifier when a class has become irrelevant or add a new binary classifier in case a new class comes up. This approach would take significantly more time in the inference stage (training stage is not critical) as inference would need to be performed for each class separately. Unfortunately, I am also facing tight time constraints at the inference stage.

Therefore, I was wondering whether (A) my idea generally makes sense and (B) if there is a more elegant/efficient multi-class classification approach (i.e. a single inference step) that is applicable in my very special case.